Monetary policy announcements of major central banks trigger substantial discussions about the policy on social media. In this paper, we use machine learning tools to identify Twitter messages related to monetary policy in a short-time window around the release of policy decisions of three major central banks, namely the ECB, the US Fed and the Bank of England. We then build an hourly measure of similarity between the tweets about monetary policy and the text of policy announcements that can be used to evaluate both the ex-ante predictability and the ex-post credibility of the announcement. We show that large differences in similarity are associated with a higher stock market and sovereign yield volatility, particularly around ECB press conferences. Our results also show a strong link between changes in similarity and asset price returns for the ECB, but less so for the Fed or the Bank of England.