Funding for Agri-food Data Canada is provided in part by the Canada First Research Excellence Fund
Collaboration
The Case for Uploadable Form Data: A More Flexible Approach to Online Submissions
Anyone who has worked extensively with online submission systems will recognize a familiar frustration: you have done the hard work of gathering, drafting, and refining your content – often collaboratively, across multiple documents and tools – and now you face the tedious task of manually copying everything into a web form, field by field. The content is ready; the process of getting it into the system is not.
This challenge comes up repeatedly in our work with the Agri-food Data Canada (ADC) and the Climate-Smart Data Collaboration Centre (CS-DCC), and it is not unique to any one platform or domain. It reflects a structural gap in how most online forms are designed: they are built for data entry, not data transfer.
The Collaboration Problem
This tension is particularly well illustrated in the context of data management plans. As noted in a recent article from Upstream:
“Data management planning is often a collaborative process, involving researchers, librarians, and institutional support staff. External tools and shared documents make it easier to iterate on plans, incorporate guidance, and ensure alignment with institutional policies and available resources. When plans are created directly within submission systems, that collaborative process can become more difficult.”
The same dynamic applies across many submission workflows. Researchers, teams, and support staff work best when they can iterate freely in shared documents, draw on existing resources, and incorporate guidance from multiple sources. Forcing that process into a single submission interface introduces friction at exactly the wrong moment – when content is nearly complete and should be easy to finalize.
Why APIs Are Not Always the Answer
The conventional infrastructure response to interoperability problems is to connect systems via APIs. While APIs are powerful and appropriate in many contexts, they come with real constraints. Both parties must be ready and willing to build and maintain the connection. Integration work requires technical resources on both sides. Security and access management become more complex. And the result is a series of point-to-point connections rather than a flexible, open approach that any participant can use.
APIs are well suited for tightly coupled systems with dedicated integration teams. They are less well suited for the diverse, distributed ecosystems that characterize research infrastructure – where institutions, tools, and workflows vary enormously and where not every participant has the capacity to build custom integrations.
A Simpler Proposal: Publishable Schemas and Uploadable Data Files
At ADC, we have been exploring a different approach. The core idea is straightforward: if an online form publishes its expected data structure as a downloadable schema or sample data file, then users can prepare their submissions outside the system – collaboratively, using whatever tools work best for them – and upload a structured file (such as a .json file) when they are ready to submit.
The form receives the uploaded file, validates it against the expected format, and populates the interface with the pre-filled content. The user retains final control, reviewing and editing within the UI before submitting. This preserves the benefits of collaborative, tool-agnostic preparation while keeping the submission process human-centered and editable.
Consider the DMP Assistant, the Alliance’s data management planning tool. Currently, users draft their plans directly in the online interface. Under this model, a researcher could instead work with their existing lab documentation, institutional guidance, and an AI assistant to compile all the relevant information into a structured .json file, then upload it into the DMP Assistant to populate the form in a single step – arriving at the editing stage with a complete draft rather than a blank form.
We Already Do This
This is not a theoretical proposal. ADC already supports this kind of workflow in the Semantic Engine’s schema development tool. We provide a structured prompt that helps users draft their schema content with an AI assistant, then upload the resulting .json file directly into the Semantic Engine editor. The file is parsed, the fields are populated, and the user continues working from there – a draft version that may contain AI errors but offers the opportunity to correct and improve.
The pattern works. It reduces manual data entry, supports collaborative preparation, and lowers the barrier for users who want to work in familiar tools before engaging with a specialized system. We believe it is a model worth broader adoption, and one that submission systems of all kinds could implement without requiring significant infrastructure investment from either side.
Written by Carly Huitema
Happy New Year and welcome back to a whole new year of blog posts by ADC!
A New Year is upon us and yet I’m still stuck questioning the value of data and how everyone else values data. I know I talked a bit about this last year in my What happened with the “old” data anyway? post and I know I have sprinkled this question in a few other posts, but I’m really struggling with the value or maybe the perceived value of TODAY’s data and not just the archived or “old” data. Are we just going to collect data as we have in the past, grab what we need from it, then let it sit on a shelf to be forgotten? We need to wake up and change the data culture, to recognize that in order to value data – we need to care for it!
Let’s see if I can explain my concerns and worries. In the agri-food industry, or really any industry, we talk about challenges at a very high level – is the industry sustainable, climate change outcomes, etc… We love to talk about the challenges, the impending changes, the effects any change may or may not have on a particular outcome, and really the list goes on….. If there is a data person involved in these conversations, you betcha that the answer ALWAYS falls back on the data to support whatever claim people are making.
BUT…..
My worry is the state of that data. I think it’s safe to say that many questions we ask – can indeed be answered with DATA. Isn’t that the basis of evidence-based decisions? But what happens if that data you are using to answer your question is NOT very well documented, is NOT stored and accessible for future use, is NOT managed or governed appropriately, or is NOT FAIR – do we have a problem?
If we recognize that data is important to help us make decisions then why are we NOT seeing resources aka funding set aside to ensure proper data management and governance? I know what you’re thinking – Michelle, you work in the research world – it’s really not your problem – so stop worrying! And that’s where I am really challenged – this problem goes beyond the research data world and I’m not getting the sense that anyone – beyond the data archivists and data managers are concerned about it! I’ve been actively participating in national data spaces meetings that cover data beyond the research field – such as industry/producer data, and in discussions such as the What is the State of Agri-Food Sustainability in Canada? CAPI Webinars – and the need for proper data management and governance beyond the research world is really becoming clear, and as an industry we are missing something crucial! We need to start seriously giving thought to supporting data management and governance at regional levels, at the associations, and at the producer level of the agri-food industry. In the research world, we can develop tools, such as the Semantic Engine and soon to be released Data Request Tracker – but we need more resources to get these out into the industry! OK let’s call that spade a spade – we need money and positions to make this a sustainable action. We need more efforts for training and discussions across the agri-food industry around the topics of data management and governance!
Oh my! Can you see I’m a tad passionate about this? If we can agree that DATA helps us make decisions – then let’s find a way to work together to better manage the data! Let’s get this data culture moving along and acknowledging the true value that data possesses.
I recently had the opportunity to conduct an in-person workshop at the Cultivating Resilience: Building Climate-Smart Food Systems Together Summit in Vancouver, BC. The Summit was hosted by the Agricultural Genomics Action Centre sister hub to the Climate-Smart Data Collaboration Centre, in which ADC is an active partner and supporter. I chose to talk about Collaboration – a topic that is near and dear to me, yet a topic that can create a lot of chaos and stress.
Collaboration can mean a few different things to people, but I have always taken the lens of people working together to achieve a common goal or to work together and exchange knowledge. In the workshop, I asked people to introduce themselves at their table and discuss their relationship with “data”. The noise level in the room rose and the conversations were fantastic! I closed this part of the workshop with the statement: “See! how easy it can be to start a collaboration?” The laughter that ensued and the audience comment: “Have you WORKED with people!! They can be… well…. interesting at times!” Yes, I agree 100% with this statement! I can make it sound so easy, yet we all know the challenges behind working and collaborating with people.
Now let’s see what “data collaboration” looks like! Let’s start with the simple example of a table:
| 1 | 13 | 74 | Sunny |
| 2 | 15 | 78 | Cloudy |
| 3 | 21 | 71 | Part cloudy |
| 4 | 28 | 99 | Rain |
| 6 | 20 | 75 | Sunny |
How in the world – can I work with this table, let alone start any collaborations? Yes – any regular readers will anticipate where I am going with this – DOCUMENTATION!!! Without it – this is garbage! Sorry folks, sometimes the truth is ugly. Doesn’t matter who, how much time, or how much money was spent on collecting the information in this table – without documentation – it’s garbage!
Think about that brief introduction chat you had at the table or with a new colleague and/or collaborator – you usually start with the basic information about yourself: your name, your occupation, where you work, and maybe something personal like city/country you live in or whether you have a pet. Now, if I had that basic information about this table – I MIGHT be able to do something with it – title? headings? It’s a start and just like any people collaboration – it needs work.
The amount of work you put into a collaboration – whether it’s people or data – can lead you down a very rewarding path and outcome. Give it a thought – especially the next time you collect data, forget to document it, and go back to use it in a month or a year – Oops!
Don’t forget the Semantic Engine a great place to start that data collaboration!
image generated by AI
