J and I and Me
MongoDB and the CAP Theorem
The CAP theorem
by Brewer basically says that a distributed systems can only have two of the following three properties:
- Consistency i.e. each node has the same data
- Availability i.e. a node will always answer queries if possible
- Partition tolerance i.e. work despite a network failure so nodes cannot communicate with one another
In real life things are a little different: You cannot really sacrifice partition tolerance. The network will eventually fail. Or nodes might fail.
So here is a different approach to understanding the CAP theorem: Imagine a cluster of nodes. Each has a replicated set of data. When the network or a node fails there are two options to answer a query: First a node can give an answer to a query based on the data on that node. This information might be outdated. The other nodes might have received some update that are not propagated yet. So the node might therefore give an incorrect answer - so the system sacrifices consistency.
The other option in such a situation is to give no answer. Then the system rather won't answer queries than give a potentially incorrect answer. This sacrifices availability - the node does not answer the query even though it is still up.
So let's take the CAP theorem to better understand MongoDB. MongoDB uses a Master / Slave replication scheme
. Data is written to a master node and then replicated to the slaves. If a network failure occurs or the master is down a slave takes over as the new master. So how does MongoDB solve the issue concerning CAP? There are settings that influence Mongos behavior:
- Write Concerns let you choose when a write attempt is considered successful. The setting vary from "error ignored" up to settings that define how many nodes must have acknowledged the write operation.
- Read Preferences allow you to choose whether you want to read from the master or also from slaves.
So concerning CAP it leaves you with different options:
- Using the write concerns you can enforce different level of consistency in the cluster - you can choose how many nodes the data must be stored at. There is a trade off with availability: The write will fail if the number of nodes the data should be stored at is higher than the currently available number of nodes.
- The read preference can be used to choose which node data should be read from. If you decide to read data from the master only you will get the data even if it has not been propagated to all nodes.
Besides a trade off between availability and consistency these settings obviously also influence performance.
If you like you can read details about what happens if a MongoDB partitions in Jepsen
So bottom line: MongoDB allows you to fine tune the trade off between consistency and availability using write concerns and read preferences. Concerning partition tolerance there is really no choice - it will eventually happen. So it can be tuned to be AP or CP or something in between - depending on how you tune it. Final note: This is my take on CAP and where MongoDB stands. If you browse around on the web you might find different takes on it. I am happy to discuss the details - leave a comment!
Labels: CAP, CAP Theorem, MongoDB
Why Java's Checked Exceptions Are an Issue
Exceptions were originally introduced to make programs safer. Back in the days languages like C used return codes to signal whether the call was actually successful. The return codes could easily be ignored. So you end up with a program that just continues executing - even though an error occured. Exceptions are meant to solve this. To signal an error an exception is thrown. This exception has to be handled if the error should be resolved. Handling the exception is separated from the rest of the control flow e.g. in a try-block. So handling an error is separated from the rest of the program.
Java introduced three types of exceptions:
- Errors like OutOfMemoryError are unrecoverable problems. So your code cannot really handle them.
- Unchecked exceptions are subclasses of RuntimeException. While they can be handled it is not mandatory to do so. Examples include the NullPointerException. This exception can be thrown any time an object is accessed. So it makes no sense to force developers to handle it.
- All other exceptions are checked exceptions i.e. the compiler ensures that they are handled using catch or the method declares that it throws the exception.
Originally I thought an exception is an extension of the return type of a method. Instead of a value of the "normal" return type it throws or "returns" an exception. So if the program should be sound concerning types every type of exception needs to be handled - just as you would handle a normal return value. Obviously it makes sense to use the type checker to ensure this happens. So checked exceptions make a lot of sense.
Checked exceptions are also used throughout the Java APIs. The developers of these APIs obviously know what they are doing - so again checked exceptions again seemed like a good idea.
However, later on I did quite a few code reviews. Some typical ways to handle checked exceptions appeared:
- Exceptions were just ignored i.e. the catch block was empty.
- Exceptions are logged. This is also what the default Eclipse code snippet does. The code just swallows the exception. However, developers are supposed to actually handle the exceptions and not just log them. More often than not a closer examination revealed that the exception really was about a serious problem and the program should probably have signaled a failure to the user - and not just continue and log the problem somewhere. This is pretty much the same situation as ignoring return codes in C - which exceptions were supposed to solve.
- Sometime the exception is wrapped and rethrown. This is more or less identical to an unchecked exception. The exception is just propagated. Just the type changes, of course - but that might not justify the code written.
Only seldom the exception is really actually handled e.g. by trying an alternative algorithm or returning some kind of default value.
You could argue that this is just the fault of the developers who wrote the code. Why did they not implement proper exception handling?
However, some facts made me wonder whether this is actually the correct explanation:
- The Spring project only uses unchecked exceptions
- Even EJB introduces the EJBException that can be used to wrap other exception in an unchecked exception.
- There is hardly any other language using checked exceptions. Wikipedia lists OCaml, CLU and Modula-3. This makes Java the only main stream language using checked exceptions. This should really make you wonder - why didn't C# for example implement this feature, too?
So apparently checked exceptions might not be such a smart idea after all - otherwise everybody would be using them. And the reason why so much exception handling is implemented poorly might be that developers are forced to write code to handle exceptions - even if there is no sensible way to do so. This is the primary difference between checked and unchecked exceptions: Unchecked exceptions have a sensible default i.e. the exception is just propagated to the next level in the call hierarchy. Checked exceptions lack such default and must therefore be handled in one way or another.
So essentially checked exceptions force the developer of the calling code to handle the exception. This makes only sense if it can be handled sensible. In a lot of cases that is not possible. Therefore I believe checked exceptions should hardly be used. In almost any case it is better to rely on an unchecked exception. Then the default is that the exception is propagated if not handled - which is usually the better alternative. Note that it is still possible to handle the exception if needed.
I think relying more on unchecked exceptions is primarily a matter of courage. A lot of projects and libraries in the Java space use checked exceptions. They are without a doubt overused - see JDBC's SQLException
for example. It is a checked exception but can hardly ever be handled sensible. Not to follow these bad examples truly takes courage. Maybe introducing checked exception is even the greatest mistake in Java's language design.
Labels: Exceptions, Java
This year's JAX
will see some interesting sessions that I would like to highlight:
- The New School Enterprise IT Day will show how new technologies and business challenges will change the Enterprise IT. I am quite happy that this has been added to the JAX schedule - because I believe there will a huge shift in this area in the next few years.
- The Cloud Computing Day will show the latest and greatest in the Cloud. Several topics - such as the different Java PaaS alternatives but also IaaS will be explained in detail.
- I have done several Code Retreats at adesso AG and always found them to be a great experience for all involved. Therefore I am pleases that I can do a Code Retreat at JAX this year.
- And of course the Advanced Spring Powerworkshop will take place - it is a unique opportunity to dive deeper into the framework.
Some of my colleagues are also presenting. For example Alexander Frommelt will talk about IT landscapes
and also about Portals and whether Portlet are really a good fit for them
. Halil-Cem Gürsoy will talk about Google App Engine
So I am really looking forward to the event - and would be glad to meet you there!
Common Misconceptions: The Waterfall Model
I think the Waterfall Model is the result of a big misunderstanding, probably one of the worst in out industry.
Look at Royce's original paper (PDF can be found here
. You will notice that the paper starts with the separation of different activities such as analysis and coding. To me that sound like an attempt to actually define basic software engineering activities instead of just unstructured hacking. The paper goes on and discusses more different phases that a project might go through. It shows a figure pretty much like the Waterfall model we are used to. No surprises so far.
But then the fun starts: The third figure already shows that the steps are not necessarily performed in order. The text says:
... as each step progresses and the design is further detailed, there is an iteration with the preceding and succeeding steps but rarely with the more remote steps in the sequence.
Let me repeat: The original Waterfall paper says that you might need to go back to previous steps, even remote ones. It even uses the term "iteration".
It goes on and discusses that once you run in production you might learn that your system does not perform well enough. That leads to major problems - and you will probably go back to the analysis. You might call it an iterative approach - even though it is probably not voluntarily.
Even better: The paper suggests:
If the computer program in question is being developed for the first time, arrange matters so that the version finally delivered to the customer for operational deployment is actually the second version in so far as critical design / operations areas are concerned.
So essentially you should do at least two iterations - the first version will not get it right. Another hint at an iterative process.
And the paper even suggests to involve the customer - probably one of the most important points in Agile practices.
Of course the paper includes sections that are quite different from the Agile school - such as the focus on documentation. But it is from 1970 - and the author is specialized in systems for spacecrafts. Those still rely a lot on documentation even today because of the extreme risks those systems have.
However, the bottom line is that the original Waterfall paper does not advocate what is now considered the Waterfall Model. It does not require going through the steps in order and it even mentions that the first release will not be a good solutions. Quite contrarily: It talks about iterations - very limited of course but it hints the direction Agile and iterative processes took later on.
I am still confused how the industry was able to misunderstand this paper. I wonder how much damage it did. Even today people still talk about Waterfall. I think everyone working with software processes should read this paper. I also suggest to use the term "Misunderstood Waterfall Model" when discussing a model that suggest going through the steps in a strict order. Because that model is just a misunderstanding, it is not what Royce described.
Oh, and next time someone talks about the Waterfall Model - don't forget to ask him or her whether he has read the original paper about it...
Some Possible Enhancements for Java and Java EE
The recent discusssion about Java EE and Spring made me think. Actually I find it odd to fight over the programming model used to develop Java applications. Neither Java EE nor Spring will go away - the market shares of both are just too large. At the end everyone involved in the discussion is trying to sell a platform i.e. an application or web server. That product must offer good support for both programming models - and all the others like Scala, Groovy etc.
IMHO the programming models have a lot in common now. Spring also covers advanced challenges like Social Media, NoSQL etc - see my older blog post Spring vs. Java EE and Why I Don't Care
. So it is not in the field of programming models where innovation is laking. Instead we should focus on improving the platform.
So what is it that would really make Java a better platform? Here are some ideas:
Ever since the invention of Java EE roles like the Application Assember were defined. I never saw a project that followed this workflow. Nowadays there is just an automated process that results in an EAR or even a complete setup including an application server. It is time to face this reality and get rid of these role descriptions. The deployment process is not done that way. Java needs to step away from it.
Nowadays application server are seldom used to host more than one application. So maybe that offers some possibilities to improve the deployment processes and also the features. If there is just one application on a server anyway the strict isolation between applications Java EE offers does not make that much sense any more.
It would be good to focus more on deployment during the development process. Applications are much more often deployed into testing or development environments than into production. I am wondering how many CPU and I/O cycles are wasted compressing WARs and JARs and then EARs during a build process - and immediately decompressing them on the web or application server so testing can start. Can't the Java EE standard do something about this? Obviously there are exploded WARs and EARs - but they are not used as often as they should be. Obviously there are tools like JRebel and Play - but what if the complete tool chain including IDEs, build tools, servers etc would support a standard here? What if all of those parties could collaborate to make this work flawlessly? I think we would see quite a boost in developer productivity.
Often I see applications modularized in several WARs. These modules need to talk to each other. However, there is no easy way to do this. Usually they end up using Web Services or another HTTP based protocol. Compared to a local method call this adds the overhead of HTTP, sockets and marshalling / demarshalling. A possible solution would be OSGi - but that technology is quite complex. The reason to me seems to be the problems around shared class loading. Classes that might represent parameters or return values can be shared between modules. JBoss includes a solution based on JMX that did not support shared classes - so modules need to have the correct classes available. This is the same as for Web Services or the other HTTP protocols. But there is no standard apart from JMX - which is very low level. Can't we have some communication between WARs that in the standard? Maybe based on JMX? Or at least best practices around it?
Concurrency in languages like Erlang is very different from what we are used to in Java. There are actors that work on a serial list of messages. Concurrency is achieved by multiple actors working in parallel - while the actors themselves are just working serially. This approach is also used by the Scala library Akka on the JVM. In Erlang this approach leads to lots of actors - far more than the number of threads that can be supported on the JVM. If the JVM wants to stay relevant for this model we need proper support for it.
Actors in Erlang and in Google Dart - they are called Isolates have separated heaps. Those heaps can be independently GC'd - so while one actors spends time doing GC the other can continue to work on messages. By design you end up with less stop-the-world-GC. Maybe we could have a similar concept in Java? This is an extension of the previous suggestion but there might be other options as well. For example each WAR file could have a separate heap.
Those are just some - possibly crazy - ideas. My intention is to focus on making the Java platform better for all of us. And I believe for that the discussion around the future of Java must focus more on the JVM and the infrastructure like web or application server than on programming models. There is quite some innovation in the area of the programming models anyway. For example there are lots and lots of Open Source projects - frameworks and complete languages. But for infrastructure standards are needed - so any server and environment can work with the solution. So that is what the standardization process should focus on.
What do you think? I am looking forward to your comments!
As you have probably noticed, the last blog posting Testing Considered Harmful
was actually a April Fool's Joke. I believe Test Driven Development is one of the most important innovations in the last years. I have been infected ever since Kent Beck explained JUnit to me.
Here are some things adesso does to make TDD a reality:
The list could go on. So please program - and drive - safely! If you were caught by the April Fool's Joke: I hope you don't mind. If you do: I apologize.
Testing Considered Harmful
Unit testing started with JUnit - and was infectious. As you can see in JUnit Test Infected: Programmers Love Writing Tests
that was the intention from the very beginning. And so the infection spread.
Nowadays almost all programmers do extensive Unit Tests. And therefore we see problems arising. Instead of thinking about a problem and building a reliable solution developers now just write a test and code away. At one point they get a green bar and consider the job done. But can you build proper software that way? Dijkstra stated in 1972: "If you want more effective programmers, you will discover that they should not waste their time debugging, they should not introduce the bugs to start with." - see Wikiquote
. I think the same holds true for testing. So instead of the ever famous code-test-refactor cycle we could use more of the Feynman Way of Solving Problems
. If you really think about the solution first you will find you won't need a test. Instead you will actually write correct code from the beginning. You won't need to rely on tests. They cannot show the absence of bugs anyway - as Dijkstra mentioned. In the end you will end up with cleaner code and less code - as there are no tests to be written.
This principle can be applied to other areas as well. I think one reason why there are so many car accidents are the very effective brakes cars have now. If we had less effective brakes people would think about possible problems beforehand, drive more carefully - and we would have less accidents.