Oracle adds long-awaited whitelisting capabilities to Java

Oracle added a feature in Java that lets companies control what specific Java applets are allowed to run on their endpoint computers, which could help them better manage Java security risks.

The new feature is called the “Deployment Rule Set” and was added in Java 7 Update 40 (Java 7u40) that was released Tuesday.

Many home users can protect themselves from attacks targeting Java by disabling the Java plug-in in their browsers or uninstalling the software completely. However, most companies can’t do this, because their employees need access to Web-based, business-critical applications that require Java support.

Many companies can’t upgrade to new Java versions for compatibility reasons, which increases the risk of their computers being compromised through Java exploits while their employees surf the Web.

A recent study by security firm Bit9 showed that over 80 percent of Java-enabled enterprise computers run Java 6, with the most widely deployed version being Java 6 Update 20. Oracle ended its public support for Java 6 in April and only users with commercial support contracts can still get security patches for it.

Security researchers criticized Oracle in the past for not adding a whitelisting feature in Java that could be used to only allow specific applets chosen by the user to run inside the browser. It seems that the company listened and the new “Deployment Rule Set” feature can now be used to do that.

The feature gives system administrators fine-grained control over the execution of applets by allowing them to create an XML file with rules for how known applets should be handled by the Java plug-in.

The rules identify Java rich Internet applications (RIAs) — Java applets and Java Web Start applications — by their location URL or title and define what action should be taken for them. The possible actions are: running them without displaying most security prompts, handling them using the plug-in’s default behavior — running them while displaying all applicable security prompts — or blocking them.

Rules added to the XML file are tested sequentially, so they can be used to create a white list. Administrators can first create rules that would match specific RIAs and allow them to run and then add a general rule at the end of the file to block all applications that don’t match the first rules.

The rule set file needs to be digitally signed with a digital certificate issued by a trusted certificate authority, packaged as a Java archive (JAR) and placed in a specific directory inside the Java installation on all computers where those rules are to be applied.

“The Deployment Rule Set feature is optional and shall only be used internally in an organization with a controlled environment,” Oracle said in the feature’s documentation. “If a JAR file that contains a rule set is distributed or made available publicly, then the certificate used to sign the rule set will be blacklisted and blocked in Java.”

Companies will need to install Java 7u40 in order to use the new feature, but the feature can be used to create rules that force certain RIAs to be executed with older Java versions installed on the same computers. This will allow companies to maintain compatibility for older applications.

It’s a great first step, but right now the level of knowledge required to create and deploy the rule sets limit the feature’s use to well-structured IT departments with good control over their systems, said Wolfgang Kandek, the CTO of vulnerability management firm Qualys. However, for an experienced system administrator deploying it should not be a problem, he said.

The feature is for managed environments, Kandek said. Home users should always upgrade to the latest version of Java, but if you are a system administrator and you need to maintain older versions of Java, then this might be an option, he said.

It’s a great mechanism for whitelisting Java Web applications, because it’s browser independent, Kandek said. Until now, companies had to use different methods for different browsers in order to do this, he said.

The functionality needs to be refined as far as management goes, but Qualys’ customers who were asked about it expressed their interest in it, Kandek said. One of those companies runs an older Java version on around 2,000 of its 12,000 machines and is worried about employees visiting malicious websites that would try to exploit that, he said.

Kandek believes that the new feature is one of several signs that things are moving in the right direction at Oracle, security wise. However, other researchers feel that Oracle should do more to fix Java’s security issues at the code level.

It’s easier to eliminate the Java plug-in as an attack vector through various GUI (graphical user interface) and policy-based “security enhancements” than to clean up the code and re-architect the platform to strengthen its security stance, Adam Gowdiak, the founder of Polish vulnerability research firm Security Explorations, said via email.

Gowdiak and his company have found many critical vulnerabilities in Java during the past two years.

When it comes to browser-based attack scenarios, this new feature should help reduce the risk associated with executing malicious Java code from untrusted sources, Gowdiak said, adding that he expects many companies to give it a try. It’s difficult to predict an actual adoption rate though and available data on Java updates in enterprise environments does not paint an optimistic picture, he said.