Simple database for tracking documents?

KatalDT

2[H]4U
Joined
Jul 28, 2010
Messages
2,567
Hi,

A local non-profit is going to start cataloging documents (paper copies) by scanning them, using OCR to convert them to PDF, then filing the hard copies away. A buddy approached me (knowing I know a bit about computers) and asked me about a simple database system.

Requirements:
Fairly easy data entry (via a web form, I'd imagine would be easy) - Volunteers would be entering the information
Fairly easy searching by any number of attributes - author, date, type of document, etc.
Sorting by category
A link to the location of the PDF for each entry
Would be nice to have a memo/text field that the contents of the document could be copy/pasted into, so they can search by content if necessary

Now, they're pretty low-cost... so the first idea of getting a small, local server isn't going to fly. I suggested they could talk to their host about a MySQL setup (or rather put me in touch with their host) and run it through that - since the PDFs will be uploaded to their host anyway. That way volunteers could also take a box of documents home and enter data from there.

Any thoughts? Is there any open source software already out there that can handle these needs? I have plenty of experience with SQL database management and PHP, but I've never built anything like this from the ground up. It would have to be very non-tech friendly to enter data and search the database though.

Thanks!
 
Last edited:
MediaWiki software could probably do those functions, uploading files and linking to them, also being able to paste in data and have it easily searchable.

Cant think of any downsides, maybe it becomes unmanageable at size?, and you would need your host to support it. Local server would make it easy.
 
MediaWiki software could probably do those functions, uploading files and linking to them, also being able to paste in data and have it easily searchable.

Cant think of any downsides, maybe it becomes unmanageable at size?, and you would need your host to support it. Local server would make it easy.

Unfortunately they're looking at *thousands* of documents... they've been storing them for some time (going back to the 1950's), so while the needs are fairly simple - one entry and it's pretty much done, nothing fancy - I'm not sure how it will scale on a Wiki.
 
How big are the files?

While you can store binary data in a database, I prefer to store the files elsewhere and only store meta data in the database. Databases just seem a very cumbersome way of managing files.
 
How big are the files?

While you can store binary data in a database, I prefer to store the files elsewhere and only store meta data in the database. Databases just seem a very cumbersome way of managing files.

Yeah, the PDFs are not going to be stored in the database directly. The way I'm seeing it right now, the PHP script that interacts with the database will link to files based on the sequential ID, relative to the location of the script. So if the script is in /var/www/docdb/, and the pdfs are in the subdirectory /var/www/docdb/pdf, it would look for the PDF (by matching the doc ID) in the relative ./pdf folder.

Just to keep it really simple. In case they ever move the server, it would be as easy as changing the config settings to access the new SQL database and looking for the PDFs in the same relative location.
 
I can't personally speak to how well a wiki scales, but wikipedia obviously still runs quite well. Or did you mean in terms of managing the wiki, not performance?
It also does have the benefit of keeping the files out of the database, they are only referenced and are stored on a filesystem in automatically organized folders, not really sure of the structure of the folders, but you can still have direct access to the files.
 
I can't personally speak to how well a wiki scales, but wikipedia obviously still runs quite well. Or did you mean in terms of managing the wiki, not performance?
It also does have the benefit of keeping the files out of the database, they are only referenced and are stored on a filesystem in automatically organized folders, not really sure of the structure of the folders, but you can still have direct access to the files.

Managing and entry. I guess entry wouldn't be very difficult - at all - to set up a template for easy data entry that converts it into a nice looking wiki page with the same information.

I will start looking at the Wiki. I've managed a few so I'm familiar with the software.

Thanks guys!
 
Anybody have any experience with DokuWiki? http://www.dokuwiki.org

Looks like it's tailored towards documents. "It has a simple but powerful syntax which makes sure the datafiles remain readable outside the Wiki and eases the creation of structured texts."
 
Have you looked into semi-structured XML databases? I believe these are used often in dealing with the type of data you are interested in. There are advantages to semi-structured databases. XML can be useful when you find you are dealing with document-centric data like books, articles, etc.

http://exist.sourceforge.net/
http://en.wikipedia.org/wiki/XML_database

You may want to look into native XML databases and related XML technologies like XPath, XQuery and XSLTs.

I can't remember which one I was looking at once, but the list of customers who used their technology was quite extensive and included a lot of research journals and organizations that mostly deal with storing documents.

When googling XML databases, some people seem overly biased towards RDBMS systems. I don't think they understand the advantages it could offer. Of course it is important to understand the differences.

I think you might want to look into it.
 
We did a proof-of-concept regarding this last year using MSSQL 2008 Filestream datatype (2008 only feature) and ADO.net. You're basically storing data as BLOBs, and is scalable with the actual storage its going to use. We used different storage tiers for our setup, tier 1 for the actual DB files, and tier 2&3 for Filestream configured data. It worked well and have implemented into production, but then again, its not a very big environment (40+ users).


P.S. Whoops, TS was looking for open-source :S Well, in any case, if you're wondering if its possible in SQL Server, there you go :)
 
Ummm...

- Use a LDAP as backend
- Write a web-based frontend to enter and retrieve data from the LDAP
 
I'd have a look at the open source version of Knowledge Tree.
Any kind of Wiki is going to be meh because it's not designed for document but for page management.
 
I'd have a look at the open source version of Knowledge Tree.
Any kind of Wiki is going to be meh because it's not designed for document but for page management.

This ^^

Sounds like the OP needs a document management system. If the organization is using the scanning software for capture and OCR, that simplifies the requirements for choosing a DMS. Based on the post - Storage/Indexing/Retrieval seem to be the main points now.


Look into:

1. KnowledgeTree - http://knowledgetree.org/Main_Page

2. LogicalDoc - http://www.logicaldoc.com

3. NuxeoDM - http://www.nuxeo.com/en/products/document-management


I've had my fair share of working with document management systems especially for indexing/retrieval. The ones that cause the most pain are those "developed" in-house (such as crazy Access/SQL dbs where repository links get broken or something goes wrong - and often the indexing process needs to be repeated :eek: )

For thousands of documents - a local server is the best (and sometimes only) choice.
If they're going to be retrieving and updating documents often, they'll also need to take into consideration the bandwidth and access limitations of their host. (Document sizes get pretty big based on scan quality).
 
This ^^

Sounds like the OP needs a document management system. If the organization is using the scanning software for capture and OCR, that simplifies the requirements for choosing a DMS. Based on the post - Storage/Indexing/Retrieval seem to be the main points now.


Look into:

1. KnowledgeTree - http://knowledgetree.org/Main_Page

2. LogicalDoc - http://www.logicaldoc.com

3. NuxeoDM - http://www.nuxeo.com/en/products/document-management


I've had my fair share of working with document management systems especially for indexing/retrieval. The ones that cause the most pain are those "developed" in-house (such as crazy Access/SQL dbs where repository links get broken or something goes wrong - and often the indexing process needs to be repeated :eek: )

For thousands of documents - a local server is the best (and sometimes only) choice.
If they're going to be retrieving and updating documents often, they'll also need to take into consideration the bandwidth and access limitations of their host. (Document sizes get pretty big based on scan quality).

Good info. Would have to be free. I'll also be trying to convince them to use a local server - maybe the HP Proliant N40L I just got for under $200 shipped could convince them. (I'd give it to them at cost, being a non-profit and all)

While researching these guys, I also ran into Alfresco - which I've heard of a number of times over the past couple years, but never investigated.

Anybody have any experience with Alfresco Community? Would it be good for their purposes? (See OP)
 
Alfresco is nice software. It's actually going more into an ECM platform - which means that it provides the potential for enterprise information management as whole. Alfresco's RM + DM management modules are top-notch. Not sure how the community version is so far, I stopped somewhere at V2~3


Based on the server you mentioned, keep in mind that some of the software listed each have their minimum recommended requirements for smooth operation:

KnowledgeTree - http://www.knowledgetree.org/Platform_Requirements

NuexoDM - http://doc.nuxeo.com/display/ADMINDOC/Hardware+and+Software+requirements

Alfresco - http://wiki.alfresco.com/wiki/Repository_Hardware


Since the software available is open-source and readily available, it may be a good idea to try them all out on an old system. Windows installations are the easiest, just download the stack and install. They're all browser based, so accessing really just requires a client with FF or IE.
 
I would suggest evernote, it is about as simple as you can get and its OCR / searching works really well.
 
(I'd give it to them at cost, being a non-profit and all)
At the danger of going OT, being non-profit doesn't mean that they have to get stuff at cost, it only means that at the end of the reporting period they cannot show any profit.

I would look into what their financial situation is and charge accordingly. I know a couple of non-profits which are totally flush with cash from grants/donations and thus have no issue paying for services just like everyone else.
 
Was just browsing around, and found the NoSQL database options. Something like CouchDB or CouchBase looks like it might do the trick with a simple PHP interface. Anybody have any experience with these?
 
Was just browsing around, and found the NoSQL database options. Something like CouchDB or CouchBase looks like it might do the trick with a simple PHP interface. Anybody have any experience with these?
Any of the suggested DB-, XML-, or flatfile-based data products can handle some simple datatypes to store the meta data about a PDF document. There are tons more that could be a good technology fit. However, I'd still lean more toward a transactional system -- if something changed, you'd at least know when it occurred and could see the change logs.

Regardless, the heavy lifting of this task is generating the PDFs themselves, and coming in second is the manual data entry grunt work for each PDF's meta data (ie: creation date, who, why, revision history, recipient, etc.). The wiki suggestion made earlier in this thread may be a good fit for loading, revising, and tracking the meta data tied to a PDF -- then the wiki page could just have a link to the PDF itself.
 
Back
Top