web stats
Channel RCS & Automatic Deployment - Mirth Community

Go Back   Mirth Community > Mirth Connect > General Discussion

Reply
 
Thread Tools Display Modes
  #1  
Old 01-11-2017, 09:37 AM
jack.downes jack.downes is offline
OBX.2 Kenobi
 
Join Date: Apr 2014
Posts: 67
jack.downes is on a distinguished road
Default Channel RCS & Automatic Deployment

Not sure where to put this, but it's helpful for me, perhaps for you?

I'm using HAProxy and Git to handle several things for me with regard to Mirth. The focus of this post is on using Git to help automatically deploy channels to multiple mirth instances. I use the same idea for the HAProxy config, and it's super simple as well.

Some context:
I have a development machine and two primary machines. I have a git repo server on campus. I'm using HAProxy with ucarp to handle the virtual IP failover. HAProxy configs and Mirth channels are git repo controlled. Using git hooks, I'm able to work on my dev machine, make a commit, and see the mirth servers load up and deploy the changed channels using the mirthcli - mccommand.

I'm still sorting out a couple clarifications, and it's really rather basic code, but I'll upload it in a bit. I'm not sure if this is totally OK with the Mirth folks, i don't wan to upload anything that could be perceived as competition to their paid products, so before I upload, could one of the forum admins let me know if it's OK to post my code for this or not?

I'm going to be taking a drive in a bit, so will be back in about 4-6 hours. Perhaps there'll be an OK or a "Please don't" by then

thanks,
Jack

Last edited by jack.downes; 01-13-2017 at 11:01 AM. Reason: Clarification
Reply With Quote
  #2  
Old 01-11-2017, 09:53 AM
narupley's Avatar
narupley narupley is offline
Mirth Employee
 
Join Date: Oct 2010
Posts: 7,124
narupley is on a distinguished road
Default

Of course you can, this is a free and open source product

Excited to see what you came up with!
__________________
Step 1: JAVA CACHE...DID YOU CLEAR ...wait, ding dong the witch is dead?

Nicholas Rupley
Work: 949-237-6069
Always include what Mirth Connect version you're working with. Also include (if applicable) the code you're using and full stacktraces for errors (use CODE tags). Posting your entire channel is helpful as well; make sure to scrub any PHI/passwords first.


- How do I foo?
- You just bar.
Reply With Quote
  #3  
Old 01-11-2017, 10:18 PM
jack.downes jack.downes is offline
OBX.2 Kenobi
 
Join Date: Apr 2014
Posts: 67
jack.downes is on a distinguished road
Default

The overall idea is shown in the attached image. What I'm doing is using Git to help with the deployment of channels and HAProxy configs. While I don't show the HAProxy configs, the idea is the same - and you probably don't care about that part... everyone has their own type of loadbalancer and opinions on it

Okay, so I'm taking advantage of the commonly used git hooks. In my case, i wanted to perform an action on a git server - everytime a channel is updated. What I want to do is to find out which channels are changed, and then load (force) and deploy said channel. In my case, I don't put into place timers because I'm well enough aware of when the traffic is busiest, and will not push my commits during the working hours!

So that it's clear, you put your hooks on into the Mirth servers' repositories. This is done on a per server basis - it's not carried out via the repo. You can't expect to put a hook in your repo and have it propagate - there's a way to do that exact thing, but this is NOT that. And I don't want that - rather, just the servers that need it, get the merge script.

Also, the developer(s) use a local Mirth instance, none of the Prod mirth instances are directly accessed. They develop on the local instance, write the channel, test it, and when happy, export to the repo - into the Test or Prod directory. When they want it loaded they push the repo up. commit as often as you want, push when you want the channels loaded and deployed.

The way I laid out my directory structure is important to my script, but it's somewhat arbitrary. In the repo called "Interfaces" I have the following directories
Code:
./HAProxy
./Mirth
./Mirth/Prod
./Mirth/Dev
./Mirth/Test
./Mirth/scripts
in the root dir (or homedir of your mirth user)
Code:
touch ~/EmptyFile
This is important for the git hook which is on the mirth engines:
Quote:
cat ~/Interfaces/,git/hooks/post-merge
Code:
#!/usr/bin/bash

# Git hook to run command after git pull

channels="$(git diff-tree -r --name-only --no-commit-id ORIG_HEAD HEAD | grep Mirth)"

check_mirth() {
  if [ "$channels" ]; then
    cp /root/EmptyFile /root/LoadChanges.txt
    /root/interfaces/Mirth/scripts/load_channels.pl
    cd /opt/mirthcli/; ./mccommand -s /root/LoadChanges.txt
    
  fi
}

check_mirth
Good, so what this does is to look for changes between now and the previous iteration of the repo, looking for changes in the Mirth subfolder. If changes are found, the script load_channels.pl is called.
I can safely call mccommand -s with an empty file - because it just logs in and disconnects without doing anything. So this is safe to do. If the file Loadchanges.txt has entries, these entries are followed.

Now the perl script which is called to perform the work:
Code:
[root@mirth3 ~/interfaces/Mirth/scripts]# cat load_channels.pl 
#!/usr/bin/perl

use Modern::Perl;
use Data::Dumper;

chomp (my @gitdiffTest = `git diff-tree -r --name-only --no-commit-id ORIG_HEAD HEAD | grep "Mirth/Test"`);
chomp (my @gitdiffProd = `git diff-tree -r --name-only --no-commit-id ORIG_HEAD HEAD | grep "Mirth/Prod"`);
my @gitdiff = (@gitdiffTest, @gitdiffProd);
my $mirthTestC = '/root/interfaces/Mirth/Test/*';
my $mirthProdC = '/root/interfaces/Mirth/Prod/*';
my $mirthChangedScript = '/root/LoadChanges.txt';
my $mirthLoadAllScript = '/root/LoadAll.txt';
my @files = (< $mirthTestC >,< $mirthProdC >) ;
my %channels;
my %changedChannels;

open(my $fh, '>', $mirthLoadAllScript)
  or die "Failed to open $mirthLoadAllScript.";

foreach my $channelPath (@files) {
   my @pathParts = split('/',$channelPath);
   my ($base, $ext) = split(/\./,$pathParts[-1]);
   
   $channels{$base} = $channelPath;
   
}
foreach my $chName  (keys(%channels)) {
  print $fh 'import "'.$channels{$chName}.'" force'."\n";
  print $fh 'channel deploy "'.$chName.'"'."\n";
}
close $fh;

# Only keep the changed

foreach my $modification (@gitdiff) {
  my @pathParts = split('/',$modification);
  my ($base, $ext) = split(/\./,$pathParts[-1]); 
  $changedChannels{$base} = $channels{$base};
}

open($fh, '>', $mirthChangedScript)
  or die "Failed to open $mirthChangedScript.";

foreach my $chName  (keys(%changedChannels)) {
  print $fh 'import "'.$changedChannels{$chName}.'" force'."\n";
  print $fh 'channel deploy "'.$chName.'"'."\n";
}

close $fh;
You can see I'm doing this as root - this is perhaps poor behavior, but... the only purpose of this machine is to run Mirth, and it's not exposed to the internet nor the network at large, it's behind a LB on a separate private network... so... I'm ignoring security for convenience.

Anyway, the code creates two files - LoadAll.txt and LoadChanges.txt

The point of this: Load changes is run *only* when changes occur - new channels, or changes to existing channels. But if you want to deploy another server, it's somewhat convenient to have a list of all the channels ready for deployment.

So the overall steps:
Crontab on the machine (mirth3 in this case)
Code:
0,5,10,15,20,25,30,35,40,45,50,55 * * * * cd /root/interfaces; git pull
- Every 5 minutes, a pull is issued in the repo "interfaces"
- When that pull is completed, a git-hook "post-merge" is called. It checks for any changes in the Mirth subdirectory, and if found runs the load_channels.pl script. this script runs, and the git-hook "post-merge" then calls mccomand -s LoadChanges.pl which is the file careated by the load_channels.pl script.


anyway, hope this helps someone. The idea of the git hook used for this is a common one for system admins, hope it helps you in some way!
Attached Images
File Type: png SimpleLayout.png (54.2 KB, 81 views)

Last edited by jack.downes; 01-23-2017 at 09:58 PM. Reason: Fixed a bug with not having an empty file when no changes made.
Reply With Quote
  #4  
Old 08-28-2019, 03:23 PM
iamonkara iamonkara is offline
What's HL7?
 
Join Date: Apr 2016
Posts: 4
iamonkara is on a distinguished road
Smile Any problem with import channel.xml force

Hi Jack,

Thanks for sharing the script to deployed the changed channels. I am trying to the same thing using my Jenkins CI/CD process but I have concerns & questions

When importing channels the use of force option is that dangerous? I think it is, but then what are the best practices.

I don't understand Perl (being more of a Ruby guy) but it seems to me that your scriptis relying on file name to extract channel names which are then used in "channel deploy ... " command. Is that true? If it is, then you are following a file naming scheme and because file names are unique you avoid any channel name conflict .... please confirm?

I have different destination addresses for different environments i.e. for test and production it is test.dest.app:3000 and prod.dest.app:3000 respectively . Now I am unable to figure out how to specify the destination URL (for an HTTP listener) using environment variables. I have tried http://${env.DEST_HOST}/ and http://$DEST_HOST but they are not evaluated and I don't want to hard code the destination address in the channel definition.


Any help is much appreciated.

Thanks,
Onkara
Reply With Quote
  #5  
Old 08-29-2019, 05:50 AM
iamonkara iamonkara is offline
What's HL7?
 
Join Date: Apr 2016
Posts: 4
iamonkara is on a distinguished road
Default

Hi Jack,

Thanks for sharing the script to deployed the changed channels. I am trying to the same thing using my Jenkins CI/CD process but I have concerns & questions

When importing channels the use of force option is that dangerous? I think it is, but then what are the best practices.

I don't understand Perl (being more of a Ruby guy) but it seems to me that your scriptis relying on file name to extract channel names which are then used in "channel deploy ... " command. Is that true? If it is, then you are following a file naming scheme and because file names are unique you avoid any channel name conflict .... please confirm?

I have different destination addresses for different environments i.e. for test and production it is test.dest.app:3000 and prod.dest.app:3000 respectively . Now I am unable to figure out how to specify the destination URL (for an HTTP listener) using environment variables. I have tried http://${env.DEST_HOST}/ and http://$DEST_HOST but they are not evaluated and I don't want to hard code the destination address in the channel definition.


Any help is much appreciated.

Thanks,
Onkara
Reply With Quote
  #6  
Old 08-29-2019, 10:25 AM
iamonkara iamonkara is offline
What's HL7?
 
Join Date: Apr 2016
Posts: 4
iamonkara is on a distinguished road
Default

Quote:
Originally Posted by jack.downes View Post
The overall idea is shown in the attached image. What I'm doing is using Git to help with the deployment of channels and HAProxy configs. While I don't show the HAProxy configs, the idea is the same - and you probably don't care about that part... everyone has their own type of loadbalancer and opinions on it

Okay, so I'm taking advantage of the commonly used git hooks. In my case, i wanted to perform an action on a git server - everytime a channel is updated. What I want to do is to find out which channels are changed, and then load (force) and deploy said channel. In my case, I don't put into place timers because I'm well enough aware of when the traffic is busiest, and will not push my commits during the working hours!

So that it's clear, you put your hooks on into the Mirth servers' repositories. This is done on a per server basis - it's not carried out via the repo. You can't expect to put a hook in your repo and have it propagate - there's a way to do that exact thing, but this is NOT that. And I don't want that - rather, just the servers that need it, get the merge script.

Also, the developer(s) use a local Mirth instance, none of the Prod mirth instances are directly accessed. They develop on the local instance, write the channel, test it, and when happy, export to the repo - into the Test or Prod directory. When they want it loaded they push the repo up. commit as often as you want, push when you want the channels loaded and deployed.

The way I laid out my directory structure is important to my script, but it's somewhat arbitrary. In the repo called "Interfaces" I have the following directories
Code:
./HAProxy
./Mirth
./Mirth/Prod
./Mirth/Dev
./Mirth/Test
./Mirth/scripts
in the root dir (or homedir of your mirth user)
Code:
touch ~/EmptyFile
This is important for the git hook which is on the mirth engines:


Code:
#!/usr/bin/bash

# Git hook to run command after git pull

channels="$(git diff-tree -r --name-only --no-commit-id ORIG_HEAD HEAD | grep Mirth)"

check_mirth() {
  if [ "$channels" ]; then
    cp /root/EmptyFile /root/LoadChanges.txt
    /root/interfaces/Mirth/scripts/load_channels.pl
    cd /opt/mirthcli/; ./mccommand -s /root/LoadChanges.txt
    
  fi
}

check_mirth
Good, so what this does is to look for changes between now and the previous iteration of the repo, looking for changes in the Mirth subfolder. If changes are found, the script load_channels.pl is called.
I can safely call mccommand -s with an empty file - because it just logs in and disconnects without doing anything. So this is safe to do. If the file Loadchanges.txt has entries, these entries are followed.

Now the perl script which is called to perform the work:
Code:
[root@mirth3 ~/interfaces/Mirth/scripts]# cat load_channels.pl 
#!/usr/bin/perl

use Modern::Perl;
use Data::Dumper;

chomp (my @gitdiffTest = `git diff-tree -r --name-only --no-commit-id ORIG_HEAD HEAD | grep "Mirth/Test"`);
chomp (my @gitdiffProd = `git diff-tree -r --name-only --no-commit-id ORIG_HEAD HEAD | grep "Mirth/Prod"`);
my @gitdiff = (@gitdiffTest, @gitdiffProd);
my $mirthTestC = '/root/interfaces/Mirth/Test/*';
my $mirthProdC = '/root/interfaces/Mirth/Prod/*';
my $mirthChangedScript = '/root/LoadChanges.txt';
my $mirthLoadAllScript = '/root/LoadAll.txt';
my @files = (< $mirthTestC >,< $mirthProdC >) ;
my %channels;
my %changedChannels;

open(my $fh, '>', $mirthLoadAllScript)
  or die "Failed to open $mirthLoadAllScript.";

foreach my $channelPath (@files) {
   my @pathParts = split('/',$channelPath);
   my ($base, $ext) = split(/\./,$pathParts[-1]);
   
   $channels{$base} = $channelPath;
   
}
foreach my $chName  (keys(%channels)) {
  print $fh 'import "'.$channels{$chName}.'" force'."\n";
  print $fh 'channel deploy "'.$chName.'"'."\n";
}
close $fh;

# Only keep the changed

foreach my $modification (@gitdiff) {
  my @pathParts = split('/',$modification);
  my ($base, $ext) = split(/\./,$pathParts[-1]); 
  $changedChannels{$base} = $channels{$base};
}

open($fh, '>', $mirthChangedScript)
  or die "Failed to open $mirthChangedScript.";

foreach my $chName  (keys(%changedChannels)) {
  print $fh 'import "'.$changedChannels{$chName}.'" force'."\n";
  print $fh 'channel deploy "'.$chName.'"'."\n";
}

close $fh;
You can see I'm doing this as root - this is perhaps poor behavior, but... the only purpose of this machine is to run Mirth, and it's not exposed to the internet nor the network at large, it's behind a LB on a separate private network... so... I'm ignoring security for convenience.

Anyway, the code creates two files - LoadAll.txt and LoadChanges.txt

The point of this: Load changes is run *only* when changes occur - new channels, or changes to existing channels. But if you want to deploy another server, it's somewhat convenient to have a list of all the channels ready for deployment.

So the overall steps:
Crontab on the machine (mirth3 in this case)
Code:
0,5,10,15,20,25,30,35,40,45,50,55 * * * * cd /root/interfaces; git pull
- Every 5 minutes, a pull is issued in the repo "interfaces"
- When that pull is completed, a git-hook "post-merge" is called. It checks for any changes in the Mirth subdirectory, and if found runs the load_channels.pl script. this script runs, and the git-hook "post-merge" then calls mccomand -s LoadChanges.pl which is the file careated by the load_channels.pl script.


anyway, hope this helps someone. The idea of the git hook used for this is a common one for system admins, hope it helps you in some way!
Hi Jack,

Thanks for sharing the script to deployed the changed channels. I am trying to the same thing using my Jenkins CI/CD process but I have concerns & questions

When importing channels the use of force option is that dangerous? I think it is, but then what are the best practices.

I don't understand Perl (being more of a Ruby guy) but it seems to me that your scriptis relying on file name to extract channel names which are then used in "channel deploy ... " command. Is that true? If it is, then you are following a file naming scheme and because file names are unique you avoid any channel name conflict .... please confirm?

I have different destination addresses for different environments i.e. for test and production it is test.dest.app:3000 and prod.dest.app:3000 respectively . Now I am unable to figure out how to specify the destination URL (for an HTTP listener) using environment variables. I have tried http://${env.DEST_HOST}/ and http://$DEST_HOST but they are not evaluated and I don't want to hard code the destination address in the channel definition.


Any help is much appreciated.

Thanks,
Onkara
Reply With Quote
  #7  
Old 09-19-2019, 10:52 AM
jack.downes jack.downes is offline
OBX.2 Kenobi
 
Join Date: Apr 2014
Posts: 67
jack.downes is on a distinguished road
Default

Quote:
Hi Jack,

Thanks for sharing the script to deployed the changed channels. I am trying to the same thing using my Jenkins CI/CD process but I have concerns & questions

When importing channels the use of force option is that dangerous? I think it is, but then what are the best practices.

I don't understand Perl (being more of a Ruby guy) but it seems to me that your scriptis relying on file name to extract channel names which are then used in "channel deploy ... " command. Is that true? If it is, then you are following a file naming scheme and because file names are unique you avoid any channel name conflict .... please confirm?

I have different destination addresses for different environments i.e. for test and production it is test.dest.app:3000 and prod.dest.app:3000 respectively . Now I am unable to figure out how to specify the destination URL (for an HTTP listener) using environment variables. I have tried http://${env.DEST_HOST}/ and http://$DEST_HOST but they are not evaluated and I don't want to hard code the destination address in the channel definition.


Any help is much appreciated.

Thanks,
Onkara
Onkara, I apologize, I didn't think anyone else was using this method. I use Jenkins CI for a lot of my stuff as well, just have not integrated this as yet.

as to test and prod, you saw that I keep all my test interfaces in one directory, and prod in another - that's how I solved that. Secondly, for multiple servers - as in places where they are reading from file directories and whatnot, I append the server name to the end of the channel, and clone the channel for as many servers as needed, appending SRV1, SRV2 as needed. that way the correct server uses the correct interface. It's simple, but the benefit is when I'm working on the channels, it's an immediate visual cue as to which server this interface belongs on.

if you read the documentation on mccommand, force is required to OVERWRITE a channel, otherwise I was unable to update the channels. That's not my goal here, I *want* to overwrite the channel.

Quote:
I don't understand Perl (being more of a Ruby guy) but it seems to me that your scriptis relying on file name to extract channel names which are then used in "channel deploy ... " command. Is that true? If it is, then you are following a file naming scheme and because file names are unique you avoid any channel name conflict .... please confirm?
yes, there is a naming schema - I group things how I like for personal preference, and the script cares nothing for the names. the very last part of the name is anything, but if it's SRV2 or SRV1, then in my case, I have a match that looks to see if it's mirth3 or mirth4 the interface will go to. We are a small hosp, I only need two mirth engines live to run everything...

Ah, I see I've updated the code a while ago. because I saw no interest in this, I didn't bother updating the thread.
Here's my most recent code to dictate to which server the interface will go.

Code:
#!/usr/bin/perl

use Modern::Perl;
use Data::Dumper;
use Sys::Hostname;

chomp (my @gitdiffTest = `git diff-tree -r --name-only --no-commit-id ORIG_HEAD HEAD | grep "Mirth/Test"`);
chomp (my @gitdiffProd = `git diff-tree -r --name-only --no-commit-id ORIG_HEAD HEAD | grep "Mirth/Prod"`);
my @gitdiff = (@gitdiffTest, @gitdiffProd);
my $pathRoot = '/root/interfaces/Mirth/';
my $mirthTestC = $pathRoot.'Test/*';
my $mirthProdC = $pathRoot.'Prod/*';
my $mirthChangedScript = '/root/LoadChanges.txt';
my $mirthLoadAllScript = '/root/LoadAll.txt';
my @files = (< $mirthTestC >,< $mirthProdC >) ;
my %channels;
my %changedChannels;
my $hostname = hostname;

open(my $fh, '>', $mirthLoadAllScript)
  or die "Failed to open $mirthLoadAllScript.";

foreach my $channelPath (@files) {
   
   my @pathParts = split('/',$channelPath);
   my ($base, $ext) = split(/\./,$pathParts[-1]);
   if ($base =~ /_SRV1$/) {
      $channels{'SRV1'}{$base} = $channelPath;
   } elsif ($base =~ /_SRV2$/) {
      $channels{'SRV2'}{$base} = $channelPath;
   } else {
      $channels{'ALL'}{$base} = $channelPath;
   }
   
}
foreach my $deliniation  (keys(%channels)) {  # Keys will be SRV1, SRV2, ALL
   next if ($hostname eq 'mirth3') && ($deliniation eq 'SRV2');
   next if ($hostname eq 'mirth4') && ($deliniation eq 'SRV1');
   foreach my $chName (keys %{$channels{$deliniation} } ) {
      print $fh 'import "'.$channels{$deliniation}{$chName}.'" force'."\n";
      print $fh 'channel deploy "'.$chName.'"'."\n";
   }
}
close $fh;

# Only keep the changed

foreach my $modification (@gitdiff) {
  my @pathParts = split('/',$modification);
  my ($base, $ext) = split(/\./,$pathParts[-1]); 
  if ($base =~ /_SRV1$/) {
      $changedChannels{'SRV1'}{$base} = $channels{'SRV1'}{$base};
   } elsif ($base =~ /_SRV2$/) {
      $changedChannels{'SRV2'}{$base} = $channels{'SRV2'}{$base};
   } else {
      $changedChannels{'ALL'}{$base} = $channels{'ALL'}{$base};
   }

}

open($fh, '>', $mirthChangedScript)
  or die "Failed to open $mirthChangedScript.";

foreach my $deliniation  (keys(%changedChannels)) {  # Keys will be SRV1, SRV2, ALL
   next if ($hostname eq 'mirth3') && ($deliniation eq 'SRV2');
   next if ($hostname eq 'mirth4') && ($deliniation eq 'SRV1');
   foreach my $chName (keys %{$changedChannels{$deliniation} }) {
      print $fh 'import "'.$changedChannels{$deliniation}{$chName}.'" force'."\n";
      print $fh 'channel deploy "'.$chName.'"'."\n";
   }
}
close $fh;
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -8. The time now is 02:07 AM.


Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2019, vBulletin Solutions, Inc.
Mirth Corporation