I’ve started playing around with distributed processing, i.e. having multiple servers set up to manage the execution of tasks, for example to assist with some over-night batch processing.
I’ve been hunting around for a light weight messaging system to allow all of the executioners and marshalling systems to communicate amongst themselves. Initially I thought about each system self hosting with something like Owin or Nancy and then have them talk together using something like SignalR. Interestingly the proof of concept worked out fine, i.e. I could get a number of systems to talk.
I finished the proof of concept work feeling good, but there was something in the back of my mind that was telling me this wasn’t the right thing to do. So I thought about SignalR and how it worked. At the top level it will attempt to establish a communications channel between the client and the server with Web Sockets. Huh? Light-bulb moment. Sockets!! Why not have the various parts of the distributed system talk with sockets? I feel like I’m on to something with this. I should be able to define a set of ports that my system is gong to use. I can the call out from a central controlling system with UDP to the network asking for the distributed components to identify themselves. Once identified TCP can step in and provide the actually communications channel.
A couple of prototypes later and I’m back to being not so sure. It should be fine, but getting everything co-ordinated isn’t working quite as expected.
In my next post I’ll post some code, issues and possible solutions.
That’s all for now. Thanks for reading and see you next time.