There are certainly lots of systems which are basically just the controls and the hub which manages them. In some cases the hub has its own computer and software, in others (cheaper) it's just an interface that connects to a computer you supply; the programming info for those is generally available if you want to write your own software rather than using theirs.
As to which to look at -- I think that's simply too large a question and out of scope, as well as being heavily opinion-based. I've played with X10, but despite being the inventors of carrier-current control they're the lowest-quality manufacturer thereof; "cheap and cheerful", fun to play with, not (for me) a permanent solution. There are other manufacturers who have taken the X10 protocol, extended it for closed-loop control and more precise control, improved its dimmer behavior, and so on. There are also folks who are doing WiFi or similar wireless signalling, which has advantages and disadvantages compared to sending the commands over the power lines. Home automation websites/catalogs will happily point you to those suppliers and give you lots of information about what's available, what its constraints are, what controllers are available for it, and so on.