CAAP plan and the difficulty of reliable OTJ's

Recently whilst constructing the mid-year mathematics report to the board, I discovered a blip in the data from a cohort group. It seemed fairly anomalous and investigation found that it was based on the difficulty for teachers and leaders to ensure consistency of OTJ's schoolwide.
Despite a reasonable start on how to gather evidence for making OTJ's, we still have a long way to go as a community of educators to ensure that this most abstract and tricky process can be made as authentic and valid as possible.
Our journey at MBS began with supporting teachers to understand how to analyse the schoolwide data and how this is only one part of the overall puzzle we have to put together in order to see where a students levels lie.
This year we are running both an ALiM and an ALL contract and an off shoot of these very valuable programs, has been the need for me to develop a CAAP plan ( Curriculum Action and Achievement Plan).
This is still an ongoing process of revision, feedback and more revision. However it has started to generate a document which aims to support all stakeholders within our school and wider community, see what a child achieving at a certain level, will 'look' like. Using the National Standards and also the ELLP, and numeracy progressions in order to build up a big picture of the skills and abilities a child will display. This is the start to building  a deeper understanding of what a child who isn't achieving will look like, and also the structures we have in place for the different tiers of intervention.
As a school, we have spent a lot of time looking at moderation within writing and have constructed a more shared understanding of what each level of writing will look like. We have also focused a lot on the collection and analysis of mathematical assessment in order to feed this into effective planning.
However we now need to use all these areas we have been working in and start to feed what we know around maths and reading in order to see how we can deepen our understanding of how to make valid OTJ's within these areas. In order to align our abilities as a staff so that by the end of the year we are able to make reliable OTJ's on a students work, in any of the 3 areas and for students at all levels in our school.