It may seem strange, but the spaces (crlf, spaces...) between your inline block divs are what creates the spacing.
If you eliminate them, the extra space disappears. (see demo here)
While there isn't a specific explanation for this behavior, my theory is that because the characters are positioned between inline blocks, the browser interprets them as spaces, similar to how it treats block elements as inline elements like spans.
<body ng-app="testRepeat">
<div class="red" ng-controller="TestCtrl">
<div class="blue number" ng-repeat="number in array1 track by $index">{{number}}</div><div class="green number" ng-repeat="number in array2 track by $index">{{number}}</div><div class="teal number" ng-repeat="number in array3 track by $index">{{number}}</div>
</div>
</body>