security - Linux: productive sysadmins without root (securing intellectual property)?

Is there any way to make a seasoned Linux syadmin productive without giving him full root access?

This question comes from a perspective of protecting intellectual property (IP), which in my case, is entirely code and/or configuration files (i.e. small digital files that are easily copied). Our secret sauce has made us more successful than our smallish size would suggest. Likewise, we are once-bitten, twice shy from a few former unscrupulous employees (not sysadmins) who tried to steal IP. Top management's position is basically, "We trust people, but out of self-interest, cannot afford the risk of giving any one person more access than they absolutely need to do their job."

On the developer side, it's relatively easy to partition workflows and access levels such that people can be productive but only see only what they need to see. Only the top people (actual company owners) have the ability to combine all the ingredients and create the special sauce.

But I haven't been able to come up with a good way to maintain this IP secrecy on the Linux admin side. We make extensive use of GPG for code and sensitive text files... but what's to stop an admin from (for example) su'ing to a user and hopping on their tmux or GNU Screen session and seeing what they're doing?

(We also have Internet access disabled everywhere that could possibly come into contact with sensitive information. But, nothing is perfect, and there could be holes open to clever sysadmins or mistakes on the network admin side. Or even good old USB. There are of course numerous other measures in place, but those are beyond the scope of this question.)

The best I can come up with is basically using personalized accounts with sudo, similar to what is described in Multiple Linux sysadmins working as root. Specifically: no one except the company owners would actually have direct root access. Other admins would have a personalized account and the ability to sudo into root. Furthermore, remote logging would be instituted, and the logs would go to a server only the company owners could access. Seeing logging turned off would set off some kind of alerts.

A clever sysadmin could probably still find some holes in this scheme. And that aside, it's still reactive rather than proactive. The problem with our IP is such that competitors could make use of it very quickly, and cause a lot of damage in very short order.

So still better would be a mechanism that limits what the admin can do. But I recognize that this is a delicate balance (particularly in the light of troubleshooting and fixing production issues that need to be resolved right now).

I can't help but wonder how other organizations with very sensitive data manage this issue? For example, military sysadmins: how do they manage servers and data without being able to see confidential information?

Edit: In the initial posting, I meant to preemptively address the "hiring practices" comments that are starting to surface. One, this is supposed to be a technical question, and hiring practices IMO tend more towards social questions. But, two, I'll say this: I believe we do everything that's reasonable for hiring people: interview with multiple people at the firm; background and reference checks; all employees sign numerous legal documents, including one that says they've read and understood our handbook which details IP concerns in detail. Now, it's out of the scope of this question/site, but if someone can propose "perfect" hiring practices that filter out 100% of the bad actors, I'm all ears. Facts are: (1) I don't believe there is such a perfect hiring process; (2) people change - today's angel could be tomorrow's devil; (3) attempted code theft appears to be somewhat routine in this industry.

Answer

What you are talking about is known as the "Evil Sysadmin" risk. The long and short of it is:

• A sysadmin is someone who has elevated privileges


• Technically adept, to a level that would make them a good 'hacker'.


• Interacting with systems in anomalous scenarios.

The combination of these things makes it essentially impossible to stop malicious action. Even auditing becomes hard, because you have no 'normal' to compare with. (And frankly - a broken system may well have broken auditing too).

There are bunch of mitigative steps:

• Privilege separation - you can't stop a guy with root from doing anything on the system. But you can make one team responsible for networking, and another team responsible for 'operating systems' (or Unix/Windows separately).


• Limit physical access to kit to a different team, who don't get admin accounts... but take care of all the 'hands' work.


• Separate out 'desktop' and 'server' responsibility. Configure the desktop to inhibit removal of data. The desktop admins have no ability to access the sensitive, the server admins can steal it, but have to jump through hoops to get it out the building.


• Auditing to a restricted access system - syslog and event level auditing, to a relatively tamper-resistant system that they don't have privileged access to. But collecting it isn't enough, you need to monitor it - and frankly, there's a bunch of ways to 'steal' information that might not show up on an audit radar. (Poacher vs. gamekeepers)


• apply 'at rest' encryption, so data isn't stored 'in the clear', and requires a live system to access. This means that people with physical access can't access a system that's not actively being monitored, and that in an 'anomalous' scenario where a sysadmin is working on it, the data is less exposed. (e.g. if the database isn't working, the data probably isn't readable)


• Two man rule - if you're ok with your productivity being crippled, and your morale likewise. (Seriously - I've seen it done, and the persistent state of working and being watched makes it extremely difficult working conditions).


• Vet your sysadmins - various records checks may exist depending on country. (Criminal records check, you might even find you can apply for a security clearance in some cases, which will trigger vetting)


• Look after your sysadmins - the absolute last thing you want to do is tell a "trusted" person that you don't trust them. And you certainly don't want to damage morale, because that increases the chance of malicious behaviour (or 'not-quite-negligence, but a slipping in vigilance'). But pay according to responsibility as well as skill set. And consider 'perks' - that are cheaper than salary, but probably valued more. Like free coffee, or pizza once a week.


• and you can also try and apply contract conditions to inhibit it, but be wary of the above.

But pretty fundamentally - you need to accept that this is a trust thing, not a technical thing. Your sysadmins will always be potentially very dangerous to you, as a result of this perfect storm.