HIST 5702

Hello weary traveller! Welcome to the beginning of your journey through the Digital History jungle. This is a <a href="http://twinery.org/"> Twine</a> which will assist you in your journey into the wide-world of Digital History. Let me introduce you to your guides! My name is Brad, and below you will meet my colleagues Sara & Kelly. We were all once like you: uninitiated, confused, and illiterate in the ways of digital ninjutsu. But through struggle and perseverance (and little help from the Programming Historian and Professor Shawn Graham), we (more or less) overcame our ignorance and inability to learn many new and useful digital skills that aid us in our historical work. We have all come away from this struggle with the undying desire to assist “digital laymen,” people who are completely unfamiliar with the digital world, who don’t know what a “Command Prompt” is or who have never heard of “datamining.” Most tutorials out there can be extremely frustrating for digital laymen, so we wanted to offer you a more colloquial and straightforward platform for learning and for really figuring out how all of this can help you personally. Now, listen to Sara explain just what you’re getting yourself into... This twine game is useful for historians and academics who are looking to add a new skill set to their academic portfolio. The following pages will explain how digital history can help historians quickly compile, analyse, and cite large sets of data for personal or academic projects. These tutorials are intended for beginners Windows users, and anyone who wants to approach history differently and digitally. As public historians who are also learning about digital history for the first time, we have tried to conceptualise projects and professions that would benefit most from these tutorials. We would also like to stress that anyone with an interest in exploring new historical methods is also encouraged to follow along! With this Twine we will not only be discussing how to use these tools, but why you should, or should not use them. With digital history, it can be tempting to just plug in a tool to make your research easier, without stopping to think about the ways in which these tools impact your project. Digital tools do not exist objectively. Like any other aspect of historical research, assumptions and the way these tools are applied influence the results we get in our research. To use these tools properly, it is important to not only understand the steps but also understand the results. So, do you wish to proceed? [[Yes|proceed]]. [[No]].

Who are you, what can we help you with, and what do you want to do? Before you delve any further on your journey, you notice a fork in the road! Left or Right? To determine your heading, we have a few questions for you. We need to know just how familiar you really are with using your computational machine (i.e. your computer)? Have you worked from the Command Line before? Can you write in Markdown? Do you future-proof your work? If you answered no to one or all of these questions or if you just need a refresher, we recommend you turn left and visit [[Basics for Digital Historians]]. . .. ... Oh. I see you’re comfortable with your digital chops... Impressive. Well then you must have something bigger in mind for your work. Want to compile data & sources? Need to organize your bibliography or convert documents? Or maybe you want to do even more to understand your collection of data and work? If so, turn right and enter [[Projects for Digital Historians|a note about digital tools]].

Basics for Digital Historians Sara intro & offering up options below. I need enlightenment and an improved way of writing/making digital files. I want to know about the virtues of [[Future-proofing, Sustainable Authorship, and Plain Text]]. My writing is ready for a new, liberated Frontier. Show me [[Markdown]]. [[command line]] Forget this simple stuff I am ready for the next phase. Take me to [[Projects for Digital Historians]].

Projects for Digital Historians Welcome to the Projects for Digital Historians section! You (hopefully) now have your basic foundation of knowledge and are ready to apply these to your own research. If you aren’t 100% certain that you can remember everything you may need to know from the basic skills section, or if you have skipped that section, don’t worry! We will have helpful paths back to the needed skills as we go through the projects. Are you ready? [[Go back and check out basic skills first|Basics for Digital Historians]] [[I’m ready]] [[I'm done. I have tried and succeeded at all the projects and the basics. I win. I am a Master|I'm done. I have tried and succeeded at all the projects and the basics. I win. I am a Master]].

Excellent! So just a note before we start: There are so many different ways to approach a digital history project and many more tools than will be discussed here. This Twine is more of a beginning look into the types of projects that may be useful to your own research. As you learn more about digital history, you may find tools that are more suited to your research needs. This section is split up into three BROAD and INTERCONNECTED projects. They are designed that you can either start at the part that is relevant to you, or go through all of them in order, if that makes more sense for where you are in your research. Sara will be taking you through the process of using digital tools to collect large quantities of data online. I (Kelly) will be taking you through the act of taking the data you’ve collected and going through some of the ways to understand that data. Finally, Brad will be taking you through a couple tools useful for organizing your your own notes, files, and bibliographic information. So either start with Sara or click on the part of the project you are currently working on, and feel free to click back at any time you need. Start [[collecting your data with Sara|collecting data with Sara]]. [[cleaning and analysis data with me (Kelly)]] Get your [[files sorted and your bibliographies in order with Brad|bibliographic organization with Brad]].

This section will provide you with the tutorials to help you gather large quantities of data for your work. The following Twine pages will help explain WGet and SPARQL and why these programs/languages are useful for academics. As archives and museum move towards digitizing their content, large quantities of data are available online at your fingertps. But how do you quickly download hundreds of diary pages without spending hours manually downloading files to your computer? You can follow these links to find out. <li> Do you want to download an entire archive quickly? Click here to learn about [[Wget]]. <li>Would you like to learn about how to scrape data from online museum databases? Click here to learn about [[SPARQL]]. <li>Are you unfamiliar with digital history and would like a crash course on computer programming? Click to learn the [[Basics for Digital Historians]]. <li>Do you already have your data? You can now learn how to analyze it. Click here to read [[cleaning and analysis data with me (Kelly)]] <li>Are you looking to cite and reference all of your data easily? Learn [[bibliographic organization with Brad]]. <li>Or, would you like to return to the [[Start]]?

Hello! So you want to do some large scale data analysis. The first thing you will need is to get some data to examine. This section does offer a discussion of data mining, however you may also want to first check out the previous section on collecting data, where Sara will take you through tools like Wget and SPARQL. [[go back to projects|Projects for Digital Historians]] [[go to data collection|collecting data with Sara]] [[I already went through Sara’s discussion but I want to learn about data mining|data mining]] [[I have all the data I need, I want to start going through it|OpenRefine]]

Organization with Brad Well, who would have thought it would be up to me, of all people, to help you get your stuff organized and tidy? My mother is off chuckling to herself somewhere... Now, I'll be the first one to emphasize that everyone has their own method of filing, storing, and organizing. And I'll be honest with you, I'm of the mind to let you keep going about your business and letting you use your <a href="http://programminghistorian.org/lessons/preserving-your-research-data"> same old methods</a>. "You do you" as a wise, old woman once said to me. With that said, I am amazed at what you can do with just a couple of the tools at our disposal. I'm certain you will be too. Where we go from here though depends on what you're looking to improve and/or solve. You've embraced future-proofing and have been writing in Markdown. Now you need to render those .md files into something beautiful. Or maybe you just need to convert all sorts of documents. Who knows?.. Well, I know that [[Pandoc]] will help a lot in either of these cases. On the other hand, when it comes to bibliographies and organizing citations, [[Zotero]] is the way to go. Create your own effective digital library and maybe even get to know your sources better. [[Why have I only got two options? I want more...]]

So perhaps you went through Sara’s discussion of data collection but still want to know about other data collecting tools. Another handy tool you can use to take your data from the internet is a tool called data mining. In the tutorials offered by the Programming Historian, they use a specific, python based example <a href ="http://programminghistorian.org/lessons/data-mining-the-internet-archive"> with the Internet Archive </a>, so our discussion will follow that example. A note about the Internet Archive: The Internet Archives is a non-profit site that bills itself as an “internet library.” Its mission is to protect the work being done on the internet from disappearing by preserving a record of what is on the internet. It has collaborated with institutions lie the Library of Congress, the Smithsonian, the federal government here in Canada and even small institutions like Carleton University. [[go back to projects|Projects for Digital Historians]] [[What is data mining?]] [[When should I use data mining?]] [[Why should I use data mining?]] [[How do I set up data mining with the Internet archive?]] [[I already have my data, skip ahead|Keep Going data mining]]

So now, either with the help of Sara or through data mining with the Internet Archive, you should have a large pile of data at your disposal. Now that you have that data, you probably want to be able to analyse it. Before we can do this though, it can be helpful to clean up the data so that it is easier to work with. One easy tool that can help us do this is a program called OpenRefine. [[What is OpenRefine?]] [[When would it be a good time to use OpenRefine?]] [[Why should I use OpenRefine?]] [[How do I set up OpenRefine?]] [[My data is already cleaned up, skip ahead|Analysing Data]] [[Go back to list of Projects|Projects for Digital Historians]]

What is OpenRefine? OpenRefine is a program, specifically an IDT (Interactive Data Transformation Tool) that allows you to make quick tweaks and changes to a large amount of data. Once cleaned up, allows you to look through unstructured data and start making connections. [[When would it be a good time to use OpenRefine?]] [[Why should I use OpenRefine?]] [[How do I set up OpenRefine?]] [[go back to projects|Projects for Digital Historians]] [[I'm good with OpenRefine, skip ahead|Keep going]]

When would it be a good time to use OpenRefine? You may be asking yourself, I already have my data why do I need to stop and use this program before actually coming to conclusions with it? The thing is that a lot of the time, when you have a lot of data, there are going to be some minor issues with it. Sometimes there are duplicates that get picked up, sometimes there are issues with spelling, and sometimes things aren’t grouped the way you would like. Here’s an example: I was working on an exhibition proposal for a new collection at a heritage site in New Brunswick. As part of that process, my group and I did some census research to figure out who in this family would have owned the items in the collection. We quickly found, however, that the census had a couple different spellings of the name Dibblee. Sometimes it was written Dibblee, sometimes it was Diblee, and sometimes it was Dibble. They were all, at this point in time, referencing the same extended family. A program like OpenRefine allows you to take Arthur Dibblee, Arthur Dibble, and Arthur Diblee and combine them so that they are all recognized as referring to the same person, if you know that this is true. Otherwise, your data will reflect three different people named Arthur. [[Why should I use OpenRefine?]] [[What is OpenRefine?]] [[How do I set up OpenRefine?]] [[go back to projects|Projects for Digital Historians]] [[I'm good with OpenRefine|Keep going]]

Why Should I Use OpenRefine? Short Answer: OpenRefine allows you to remove duplicate records, blank records, separate data that got combined along the way and combine data that is separate even though it shouldn’t be. It also allows you to get to know your data and begin making connections by going through some of the different features (like clustering, and the various facet features). Long answer: Like anything in historical research, tools like OpenRefine don’t exist in a vacuum. To find a brief discussion of the implications of this tool [[click here|Long Answer OpenRefine]]. If you don’t really care about the implications, or just don’t have time right now feel free to keep learning about OpenRefine through your normal navigation tools. [[What is OpenRefine?]] [[How do I set up OpenRefine?]] [[When would it be a good time to use OpenRefine?]] [[go back to projects|Projects for Digital Historians]] [[I'm good with OpenRefine, skip ahead|Keep going]]

Analysing your data Now that you have cleaned up your data, with the help of OpenRefine, you can now start analysing that data. A possible problem is that you have A LOT of data and are not really sure where to start. In this project we have two possible options to begin understanding and analysing your data, MALLET and AntConc. These are definitely not the only tools you can use to look at and make connections within your data, but hopefully this will be a good introduction to some of the options and you can eventually find others that may work better for you. So first things first, are you going to use MALLET or AntConc? I would recommend at least checking out the intro to both and see which one you think would be most helpful to your research specifically. [[What is MALLET?]] [[What is AntConc?]] [[Go back to Projects|Projects for Digital Historians]]

The Implications of OpenRefine If there is an overarching lesson behind OpenRefine it would be to check your data. Don’t assume that your data is fine the way it is, but go through it so that it is cleaned up and easier to work with in the future. While doing this, however, remember that digital tools are just as subjective as any other research tool. The decisions you make at this point in your project will influence the results you get later in your analysis. To go back to my [[Dibblee|When would it be a good time to use OpenRefine?]] example from earlier, grouping the Dibblees, the Dibbles and the Diblees together may make sense for the time period you are looking at, however it can also hide the fact that the Dibbles and the Dibblees eventually consider themselves separate families after generations of that division in spelling. The way that you group categories, names, spellings etc. implies certain assumptions that you have about your data and will internalize those assumptions throughout your later analysis. Keep this is mind as you go through the rest of your project and acknowledge the assumptions inherent in your data as you would if this were being done via a more traditional approach. [[What is OpenRefine?]] [[When would it be a good time to use OpenRefine?]] [[How do I set up OpenRefine?]] [[Go back to Projects|Projects for Digital Historians]] [[I'm good with OpenRefine, skip ahead|Keep going]]

How do I set up OpenRefine? Setting up OpenRefine is fairly straightforward as far as digital tools go. Unlike some, you don’t need to use the [[command line]], or some type of python extension. Go to openrefine.org and download 2.6 windows kit. Fun fact that I learned while downloading this program: make sure you have Java installed and up-to-date. I tend to be one of those people that never updates and downloads when my computer asks me to and so I had to get Java as part of this process. (PSA- Don’t be like me, listen to your computer when it asks to update things). Once Java is installed, OpenRefine should install without issue. (For detailed instructions on downloading and using OpenRefine, check out <a href ="http://programminghistorian.org/lessons/cleaning-data-with-openrefine"> the Programming Historian tutorial </a> on the topic.) Another thing to note: Although OpenRefine opens in your preferred internet browser, it is running locally on your computer. The data is not being stored online, so keep that in mind in terms of saving and whatnot. [[What is OpenRefine?]] [[When would it be a good time to use OpenRefine?]] [[Why should I use OpenRefine?]] [[Go back to Projects|Projects for Digital Historians]] [[I'm good with OpenRefine, skip ahead|Keep going]]

So now you have OpenRefine set up… CONGRATS! You can now clean up your data, whether that is the data you got with the help of Sara, or the data you got on your own. Go through the different features, check out the Programming Historian tutorial <a href="www.programminghistorian.org/lessons/cleaning-data-with-openrefine"> Clean data with OpenRefine</a>, and experiment to figure out and better understand your data. If you are ready to start examining your data [[click here|Analysing Data]]. If you want to learn about collecting your own data, go check out [[Sara's discussion|collecting data with Sara]]. If you feel like you understand your data and want to try a different project, check out [[Brad's discussion of organizing your sources|bibliographic organization with Brad]]

What is MALLET? MALLET is one of many programs you can use for topic modelling. Topic modelling is a process that takes a text and looks for patterns within it. It makes assumptions based on relationships between words to come up with a list of topics. MALLET can be used either through a GUI (graphical user interface) or through the command line. The command line gives a more fined tuned use, with greater ability to fiddle with the parameters. If you aren’t sure about using the command line, or if you want a refresher, check out our command line discussion [[here|command line]]. [[Actually I want to use Antconc|What is AntConc?]] [[When would it be a good time to use MALLET?]] [[Why should I use MALLET?]] [[How do I set up MALLET?]] [[Go back to list of Projects|Projects for Digital Historians]] [[I'm good with MALLET, skip ahead|Keep Going MALLET]]

What is AntConc? AntConc is a standalone program you can download onto your computer in order to do large scale text-based data analysis. It is super easy and straightforward to use. Unlike the topic modelling program, MALLET, that needs you to use the command line, this program does not involve the command line, making it more user friendly for those without a lot of command line experience. [[Actually I want to use MALLET|What is MALLET?]] [[When would it be a good time to use AntConc?]] [[Why should I use AntConc?]] [[How do I set up AntConc?]] [[go back to lists of Projects|Projects for Digital Historians]]

Welcome to a brief introduction on what the command line is and how to use it. In this section, we will go through some of the key things to know when using the command line, as well as some ways you will need to use it through the other discussions. Bonus Tip before starting: If you haven't started doing this already, stop including spaces in the names of your files. DON'T DO IT! Command lines don't like spaces in titles. Save yourself the headaches and start using "-" to break up words in your file names, rather than spaces. [[go to projects|Projects for Digital Historians]] [[go to basic skills|Basics for Digital Historians]] [[What is the command line?]] [[When should I use the command line?]] [[Why should I use the command line?]] [[How do I get the command line on my computer?]]

When would it be a good time to use MALLET? It would be a good time to use MALLET if you have a large number of documents. Although you could use it for a couple of documents, the key with topic modelling is that it can go through hundreds of documents that would take forever going through by hand and pull out patterns that you may miss. You’ll need to make sure that the files you want to go through are saved in a single folder in your c:\mallet folder. They all need to be in .txt format and you’ll need to have them split up for how you want to look at them. For example, if you want to look at things at the level of individual paragraphs, save each paragraph as a separate file. [[go back to projects|Projects for Digital Historians]] [[Why should I use MALLET?]] [[What is MALLET?]] [[How do I set up MALLET?]] [[I'm good with MALLET, skip ahead|Keep Going MALLET]]

Why Should I use MALLET? Short Answer: You should use MALLET if you are interested in finding patterns that you may not have seen on your own. It helps you to decipher themes in your research. Long Answer: To see a more detailed discussion of the strengths and weaknesses of MALLET, [[click here|long answer MALLET]]. If you don’t really care about the implications of this tool, that’s fine, feel free to keep moving on. [[What is AntConc?]] [[When would it be a good time to use MALLET?]] [[How do I set up MALLET?]] [[Go back to projects|Projects for Digital Historians]] [[I'm good with MALLET|Keep Going MALLET]]

How do I set up MALLET? To install MALLET, go to mallet.cs.umas.edu/index.php. I installed MALLET 2.0.7 (there is a 2.0.8 model but I haven’t used it). Once you have downloaded the file, unzip it into your c:\drive so that you can find it easily in the command line. Also make sure you have installed the Java developer’s kit. You’ll need to go into your computer, into systems, and click “advance systems settings”. Type “MALLET_HOME” into the “variable name” box, and type the path of your MALLET file (in the c:\ drive). This will tell the computer where to find the components of MALLET. Once you’ve done this, you can run MALLET through the command line. There are 9 MALLET specific commands that you can use. They are listed in MALLET and you can find them through the command `bin\mallet`. BONUS TIP: When I originally learned how to use MALLET I accidentally unzipped the file within another file so that the location of the program was actually c:\mallet\mallet. Make sure you don’t do this because your computer will not be able to find it and it will be frustrating… trust me. For a detailed tutorial regarding MALLET, <a href ="http://programminghistorian.org/lessons/topic-modeling-and-mallet"> click here </a>. [[go back to projects|Projects for Digital Historians]] [[What is MALLET?]] [[When would it be a good time to use MALLET?]] [[Why should I use MALLET?]] [[I'm good with MALLET, skip ahead|Keep Going MALLET]]

The Implications of MALLET Thank you for taking a second to think about the implications of a tool like MALLET. While the tools we’ve been discussing have been useful, it’s important to be reflective about how they impact our research. The thing about MALLET is that, while it is a really useful tool, it can sometimes be used without criticism. MALLET, at its most basic, is a simulation. You give the computer a set of parameters and it runs a simulation based on how many topics you are expecting it to have. It is also impossible to know when you start how many topics you should have. Whatever number of topics you tell your computer, which is going to influence your results, thereby affecting your research. Something else that will affect your results is the removal of stopwords. Stopwords are words like “the, it, is, to, as” etc. These words are perceived as not containing significance to the search and tend to come up a lot, which can therefore influence your topic modelling. When you take these words out, however, you are changing the way that the computer will read your data, and can cause you to miss interesting connections with these words. While many people do remove them, and it is fine to do so, it is important to acknowledge how this will influence your results and to be transparent in their removal. Listing the stopwords removed as part of your methodology is one way to do this. [[What is MALLET?]] [[When would it be a good time to use MALLET?]] [[How do I set up MALLET?]] [[Go to Projects|Projects for Digital Historians]] [[I'm good with MALLET, skip ahead|Keep Going MALLET]]

You have now learned about MALLET and how to use Topic Modelling to better understand your data! If you are still curious about the opportunities available using other tools, check out our [[AntConc|What is AntConc?]] discussion. If you realize that you do not have data that has been cleaned up enough, check out the [[OpenRefine]] discussion. If you STILL don’t have any data, look at [[the data collection project|collecting data with Sara]]. If you are tired of me asking these questions and want to move on to a different set of skills, I suggest that you take a look at [[Brad’s|bibliographic organization with Brad]] discussion on the organization of your research.

Future-proofing, Sustainable Authorship, and why you should use plain text instead of MS Word Writing, storing, and retrieving documents play a huge part in your everyday work as an historian, researcher, student, millennial, digital native, etc., etc. Yet most of us use digital tools and file formats that can be fragile in terms of their future usability, such as <a href="http://www.antipope.org/charlie/blog-static/2013/10/why-microsoft-word-must-die.html"> Microsoft Word</a>. I have used Microsoft since almost before I can remember... You probably use it for almost all your writing, like me… But you wanna know [[what]]?

With constant changes in operating systems, file formats, and programs/tools – most of which are actually designed to become obsolete in almost no time at all – creating work that you will be able to save and use far into the future can be surprisingly tough. Being stuck with tools like Word can be frustrating when new versions are forced on us and make our older tools and formats obsolete. Slowly but surely we are obligated to update more and more often. But using these tools can also be dangerous in the long term: behind a Microsoft Word or Google Docs document are incomprehensible codes that only machines understand. Only specific licensed tools, owned by Microsoft, Google, etc. can read these codes and make your .docx and .pdf files readable to a human being. Eventually, as these companies phase out the older tools & formats, our older documents become difficult to open, read, search, print, and convert into other file formats because we no longer have access to the software that can open them. Think of the whole .doc versus .docx <a href="http://en.nothingisreal.com/wiki/Please_don't_send_me_Microsoft_Word_documents"> debacle</a>. [[Oh no! What do I do? How do I save future-me from certain destruction?]] [[I’m not afraid...]]

This is where the concept of future-proofing becomes important. Ensuring that you are independent from these tools can save you a lot of trouble in the future. To do this, you should consider writing, saving, and storing your work in plain text, using simple text editors such as <a href="https://www.sublimetext.com/3"> Sublime Text 3</a> or <a href="http://www.vim.org/"> Vim</a>. (I personally use Sublime on my Windows machine. I suggest downloading it right quick, because you’ll need it for all the fun things you will be learning in this Twine). Plain text has been around for a very long time and files written this way – whether they were created twenty years ago or twenty days ago and even if the editor you are currently using stops working – continue to be near-universally readable and usable. But not only should you be writing in a plain text editor, you should also be using <a href="<a href="http://daringfireball.net/projects/markdown/"> Markdown</a>. To start learning how you can abandon Microsoft & Google, liberate yourself and your work, and begin future-proofing, [[explore Markdown|Markdown]]. If you’re already familiar with Markdown, want to continue the good fight of effectively protecting and sharing your work, and just need to convert your plain text files into different file formats, why don’t you head over to the [[Projects for Digital Historians]] section on [[Pandoc]]? Back to [[Basics for Digital Historians]]. . .. ... Hey! Down here! Want to hear some thing [[ironic]]?

Fine... You think you're all that? Go ahead and try the [[big leagues|Projects for Digital Historians]] then. . .. ... Oh, you're not ready? Ha! I thought so. Back to [[basics|Basics for Digital Historians]] with you!

Markdown Have you ever wanted to learn how to write in a digital syntax? Do you want the key to writing and creating research in a sustainable way that is independent of the litany of word-processing programs on the market? Would you prefer to embed the formatting of your work (font, layout, structure, etc.) within the text? Well, if so, Markdown is here for YOU! And if you don’t have any of these problems, Markdown is STILL here for YOU! [[What is Markdown]]? [[When and why would you use Markdown]]? [[How do I learn Markdown]]? [[Go back|Basics for Digital Historians]]. Move on the [[Projects for Digital Historians section|Projects for Digital Historians]] to continue learning how you can [[future-proof|Future-proofing, Sustainable Authorship, and Plain Text]] your work and [[convert your work written in Markdown into other file formats|Pandoc]].

Pandoc Do you [[future-proof|Future-proofing, Sustainable Authorship, and Plain Text]] your work? Do you need to convert that article written in [[Markdown]] into a Word document (.doc or .docx), an Adobe readable document (.pdf), or another file format? Or maybe you’d just like to convert documents from one file format to another (is this actually possible)? Well look no further, because Pandoc is here to help! [[What is Pandoc]]? [[When and why would you use Pandoc]]? [[How do I download Pandoc]]? [[How do I learn to use Pandoc]]? [[Go back|Projects for Digital Historians]]. Not really sure you can use Pandoc? Not familiar with Markdown or future-proofing? Then what are you doing here?! Get over to the [[Basics for Digital Historians section|Basics for Digital Historians]]!!! [[I'm done. I have tried and succeeded at all the projects and the basics. I win. I am a Master|Pandoc end]].

I wrote all of this in Microsoft Word… I am not future-proofing the majority of my work… I don’t often write in Markdown… I am hypocrite, hear me roar! Future-proofing is not for everyone. As a creature of habit I find it really difficult to let go of Word and .docx. I’ve been using it for the past decade after all… The power and responsibility that comes with making the switch can be a little daunting too. So even though I truly believe in the need to abandon Word and to strive for sustainable digital work habits, I have not transitioned to plain text or Markdown. If you’re still not personally convinced, go visit the Programming Historian’s Sustainable Authorship <a href="http://programminghistorian.org/lessons/sustainable-authorship-in-plain-text-using-pandoc-and-markdown"> page</a>. It’s where I got most of my knowledge on the topic. I like to think of the transition kind of like a diet... I know I should eat healthy, I would love to eat healthy, I know how to eat healthy, BUT I don’t. Because it requires responsibility, commitment, time, and perseverance. But I owe it to myself to try… Do you? You have another option on your hands, my friend… [[Take the diet, eat the broccoli, and follow the road less travelled|Markdown]]. OR [[Keep eating the chocolate cake and give up on the idea|Basics for Digital Historians]].

What is Markdown? Markdown is an extremely simple [[plain text|Future-proofing, Sustainable Authorship, and Plain Text]]-based syntax (a set of rules, principles, and processes that govern the structure of sentences in a language). It is a writing convention that allows you to make formatting and other changes in your writing that are independent from the program/text editor you are using to write with and that are embedded in your writing. For example, instead of writing “Brad” and pushing the BOLD button – like you would in Word – to make it “Brad” you can instead write “``**Brad**``”.This goes for structuring the text with titles, sections, headers, bullet points, indented paragraphs, web-links, illustrations, and bibliographic notes and references. So instead of having a machine, program, and code register your formatting choices for you, you are personally in charge and responsible when using Markdown. Not just that, but it is also designed to be as human-readable as possible, unlike other digital syntaxes/languages/codes like HTML. Markdown can be written in any plain text editor such as <a href="https://www.sublimetext.com/3"> Sublime Text 3</a> or <a href="http://www.vim.org/"> Vim</a>. (I personally use Sublime on my Windows machine. I suggest downloading it right quick, you’ll need it). Your plain text files written in Markdown are then saved as .md files which can be converted into any number of file formats, like .docx and .pdf. It is once these files are converted that your “``**Brad**``” will transform into “**Brad**”. But [[when and why would you use it|When and why would you use Markdown]]? For more info on Markdown, visit the creator’s <a href="http://daringfireball.net/projects/markdown/"> homepage</a> for the syntax (along with a conversion tool to transform your Markdown into HTML). [[Go back|Markdown]].

When and why would you use Markdown? Why Markdown, you ask? Well if you haven’t been over to the [[future-proofing your work section|Future-proofing, Sustainable Authorship, and Plain Text]], check it out first. It will tell you why you should [[almost|ironic]] always be writing and saving your work as plain text files. Did you go an check? [[No|Future-proofing, Sustainable Authorship, and Plain Text]]? [[Yes]]?

How do I learn Markdown? How do you learn Markdown? With PRACTICE, of course! I can tell you from personal experience that it’s one of those things that you get better at the more you do it. I have collected a range of quick tutorials and tools that have helped me learn the basics and that have directly inspired this Twine. Only by practicing the different commands will you master it. So if I were you, I would start off by playing around with <a href="http://dillinger.io/"> Dillinger</a> first. This is an online Markdown editor that you can use in your web browser. Dillinger is extremely helpful not only because it offers very useful tips on how to use Markdown, but because it also allows you to see what your writing in Markdown would look like once it was converted to show your formatting. For example, in the Markdown window I would write “``*Brad*``” while in the other window it would look like “*Brad*”. Dillinger is kind of like Markdown training wheels… Forget training wheels, I want to straight up <a href="https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet"> cheat</a>. Lay some [[more resources]] on me. [[Go back|Basics for Digital Historians]]. Move on the [[Projects for Digital Historians section|Projects for Digital Historians]] to continue learning how you can [[future-proof|Future-proofing, Sustainable Authorship, and Plain Text]] your work and [[convert your work written in Markdown into other file formats|Pandoc]].

What is Python? Python will be an integral component of your work as it is a computer programming language that runs on Mac, Linux, Windows and other operating systems, and is integrated into your [[command line]]. In order to run python script on your computer, you will need to download the language online <a href="https://www.python.org/downloads">click here to do so</a>. Downloading python allows you to access the python interpreter. The python interpreter essentially allows your computer to understand the python language and run commands. In order to make sure it is properly installed on your computer click to read this tutorial titled [[Where is Python?]]. In order to make sure you are running in python you will need to make sure your command prompt shows >>> whenever you are entering code. If you are in the command prompt, all you need to do to access the interpreter is type python and click enter. If python is installed correctly, you should automatically see >>> when typing. [[What is the command line?]] [[How do I get the command line on my computer?]] [[When should I use the command line?]] [[go back to basic skills|Basics for Digital Historians]]

Tutorial: Where is Python? • Unable to find python? • My computer was unable to find python (after installing it a number of times, and including me moving it to my desktop, etc.) • Dr. Graham helped me re-install the program. The problem was that Windows was not finding python in any of the downloading locations. Here’s how we fixed that: 1. Open up control panel. Click on “System”, “advanced system settings” and then “environmental variables” 2. This opens up two boxes. Move to the bottom box and find the PATH variable and click on it. 3. Once highlighted, click on edit, highlight everything in the box, and then paste it somewhere where you can read it. If you don’t see anything that says Python in that pasted text, then your computer is not recognizing it as a program it has… so we need to fix that! 4. If you have downloaded Python already, go to the original file you have (or if you haven’t installed it, install it until you see the containing folder of whichever version you are using (for example, I had a Python27 folder after opening it up and installing it) 5. Your python folder needs to now be moved into C:\Windows – which should be on the left hand side of your screen above RECOVERY (D:) 6. Drag and drop your Python folder into this area. If you have the command line or powershell open, this may affect installation – so close these off and try again if you get an error message 7. Once installed, click on Windows (C:) and locate your file – if it’s there – great, you’ve done the right job 8. You must now go back and repeat step 1. Find the Path variable again, click on highlight and edit. 9. At the end of the highlighted text after the last \, type in ;C:\Python27 10. Click done and exit out of the environmental variables tab. You now want to open powershell or command line and see if your computer now recognized the program. 11. Type in [python -V] into powershell and hit enter. This will now indicate what version you have. It should work. 12. If wishing to install pip.py, you can now type in [python pip install pymarc] or, if that does not work, [python -m pip install pymarc] 13. It should install and your problems should be fixed! [[Start]] [[What is Python?]] [[command line]] [[Basics for Digital Historians]] [[Projects for Digital Historians]]

As plain text, Markdown files are very flexible compared with other formats like Word. Plain text has been around for a very long time and files written this way – whether they were created twenty years ago or twenty days ago and even if the plain text editor you are currently using stops working – continue to be near-universally readable and usable. Thus, Markdown plain text files can replace Microsoft or Apple software that has many problems and that will become out-dated relatively quickly. Writing in this way liberates the author, the writing, and the formatting from the tool. Meaning your plain text file (saved as a .md file) isn’t stuck to that <a href="https://www.sublimetext.com/3"> Sublime Text editor</a> you’ve been using to write; any number of programs can open and edit your files now (unlike Word documents that need to be opened by other Word programs). There is power in this independence! You like power, don’t you? [[Yes, I like power]]. [[No, I don’t like it|ironic]].

GOOD! Because everything you write and author – your notes, articles, blog posts, wikis, syllabi, recommendation letters, etc. – can all be written in Markdown. And they can all be published on a multitude of platforms and in a variety of formats. As Digital Historians we’ve got projection slides, paper printing, web-based articles and blogs, and even mobile to think about. Being able to easily publish the same piece of research or work across all of these platforms/formats is essential and plain text allows you to do this. [[Give me an example]]. I’m convinced, [[teach me how to get down and Markdown|How do I learn Markdown]]!

Let’s say you have a single piece of work written in Markdown. Pretend it is a short history of the Mouse pad. First off, Markdown supports your use of the typical academic writing accessories: footnotes, tables & figures, and bibliographies. You can also easily convert your plain text, Markdown document on the history of the Mouse pad into a .pdf for quick dissemination. Or convert it into HTML for the web. Or you can submit your file to a digital news outlet who can easily publish your work with little effort required on their part for formatting, etc. As Markdown continues to quickly grow as a digital writing convention, more and more digital publishing platforms support Markdown, like <a href="https://wordpress.com/"> WordPress</a> and <a href="https://www.reddit.com/"> Reddit</a>. [[Soooo]]??

Soooo, if you like sustainability, ease of access and dissemination, independence from the corporate masters of the digital realm, and control of your work among many other things, Markdown is for you. On the other hand, if you want to avoid learning new things, taking on a brave new world, facing the final frontier, and are just sort of apprehensive about writing Plain Text files with an easy-to-use digital syntax, so be it... BUT, if you're willing to break the Google/Microsoft habit, put in the extra effort, and take that plunge, it will be worth it... Honestly, you will benefit in the long term if you employ Markdown across your work and the types of documents you create. It will be tough; it will be frustrating. But saving, searching, sharing, and organizing your materials - and so much more - becomes much simpler. So, to answer your original question, you should be using Markdown ALL. THE. TIME. But [[how do I learn to use Markdown|How do I learn Markdown]]? I think this is stupid, not worth my time, and I want to do [[something different|Projects for Digital Historians]]... Tell me more about how you, Brad, can have such <a href="https://github.com/BradWiebe/HIST5702-Notes"> cognitive dissonance</a> over Markdown and Digital History in general. [[Go back to Basics|Basics for Digital Historians]].

We get it, it's tough. On many occasions Sara, Kelly, and I have pulled our hair out at how frustrating "going Digital" can be. The tools are difficult to use, evil corporations are out to get you, the digital is revolutionizing space, time, society, culture, philosophy, and everything in between. After being chewed up and spit out by "HIST5702W Digital History" we still aren't sure if we still "[[wish to proceed|proceed]]." But you know what? It's time to pony up. Get your act into gear and click [[proceed]] you <a href="http://www.urbandictionary.com/define.php?term=luddite&defid=7017291"> Luddite</a>!

Zotero Ahh! I see that you’ve collected a bunch of sources there! They’re all over the place you say? You need help sorting and organizing them? Well fear not! Zotero is here for you! Before us Windows-users get started with the more intensive aspects of Zotero though, you will need to be familiar with [[Command Line|command line]], [[Python|What is Python?]], and [[Scripting|When should I use the command line?]]. You'll also need a <a href="https://www.sublimetext.com/3"> plain text editor</a>. Don't worry about all this now if you don't want because I will be pestering you about it plenty... [[What is Zotero]]? [[When and why would you use Zotero]]? [[How do I download Zotero]]? [[How do I learn to use Zotero]]? [[Go back|Projects for Digital Historians]]. [[I'm done. I have tried and succeeded at all the projects and the basics. I win. I am a Master|Zotero end]].

What is Pandoc? Pandoc is a [[command line]] tool that converts your [[plain text files|Future-proofing, Sustainable Authorship, and Plain Text]] (like your .md [[Markdown]] files) into a number of formatted file types: PDF, .docx, HTML, LaTeX, slide decks, and many more (probably some you've never even heard of). Pretty straight forward huh? Probably pretty useful too. Let's check it out!!! [[When and why would I use Pandoc|When and why would you use Pandoc]]? [[How do I download Pandoc]]? [[How do I learn to use Pandoc]]? For more on Pandoc, visit the Programming Historian’s <a href="http://programminghistorian.org/lessons/sustainable-authorship-in-plain-text-using-pandoc-and-markdown"> tutorial</a> or Pandoc’s <a href="http://pandoc.org/"> website</a>. [[Go back to Projects|Projects for Digital Historians]].

When and why would you use Pandoc? Well, before you even consider using it, you'll need a <a href="https://www.sublimetext.com/3"> plain text editor</a> and some familiarity with the [[command line]]. But really, why would you use it? To be completely honest, you would most likely use Pandoc for some simple file format conversions. With so many different file formats out there, we are often called on by professors, publishers, etc. to submit specific types of digital document types. Sometimes our work is in a file type that does not match what is required of us. Pandoc could potentially assist you in this regard. Very often I've needed to convert numerous files to .pdf with little time to spare. The free web-convertors out there were never usually up to the task, whereas using Pandoc has really helped. Ok, [[butttttttttt]]...

How do I learn to use Pandoc? This will be a little tricky. Can you handle that? Sure ya can, you've made it this far.. Alrighty, first off you will definitely need to know how to use the [[command line]] for all of this to work. And a text editor. I'm not even going to link to one, because if you're reading this you should already have passed a point where I would have told you to download one. You had also better have Pandoc on your computer already! I will be really disappointed if you're trying to get this done [[without it|How do I download Pandoc]]... I want to [[get'er dunn|get this done]]!

How do I download Pandoc? It's actually really straight forward. Just go to Pandoc's <a href="http://pandoc.org/installing.html"> install page</a>. If you're on Windows it will bring you <a href="https://github.com/jgm/pandoc/releases/tag/1.17.0.2"> here</a> and you will simply have to click on the "pandoc-1.17.0.2-windows.msi" download. Follow the instructions from there. . .. ... Now, while I've got your attention and you're busy downloading, I should let you know that you'll also need a plain text editor if you don't already have one (which you most certainly should by this point). I use <a href="https://www.sublimetext.com/3"> Sublime Text 3</a> on my Windows computer, but you could always try <a href="http://www.vim.org/"> Vim</a> or whatever. . .. ... Ok, you've finally got it installed on your computer? Get a move on because using this thing effectively isn't immediately obvious. [[How do I learn to use Pandoc]]? I'm not really sure what I'm doing here and [[what the hell this is even for|When and why would you use Pandoc]]...

What is Zotero? Zotero is a program used for collecting, storing, and organizing content, namely bibliographies. Think of all the sources, citations, references, etc. you've used or stored on your computer. Zotero deals with keeping their bibliographic information (author, year of publication, title, etc.) sorted. Zotero can be used from the program's interface itself or, as you'll see, from the [[command line]]. After bibliographic data is inputted Zotero can also export the info as a BibTeX file (essentially just a plain text file of all the inputted citations). Bibliographic references can be put in manually by hand, using our bibliography as a template But [[when and why would you use Zotero|When and why would you use Zotero]]? [[How do I download Zotero]]? [[How do I learn to use Zotero]]? For more info, visit the Zotero <a href="https://www.zotero.org/"> homepage</a>.

When and why would you use Zotero? If you've got primary/secondary sources, citations, and/or whole libraries of content whose bibliographic information needs to be collected in one easily accessible place, organized for ease of reference when writing/researching, and even excavated for information if you're really ambitious and plan on sticking around for the long haul, you should be using Zotero. Now this might not sound very impressive. Maybe you have your own system of keeping this information tidy and handy for your work. As I said before, "you do you." But you won't know if it's a better method unless you try it. So I'm not gonna waste more of your time selling this one to you, you should just try it already... Alright, fine, [[how do I download Zotero|How do I download Zotero]]? Cool. I've got Zotero and I want to [[learn how to use it|How do I learn to use Zotero]]?

How do I download Zotero? You've actually got a couple of options here... You can visit the <a href="https://www.zotero.org/"> Zotero website</a> itself and simply <a href="https://www.zotero.org/download/"> download it</a>. The Programming Historian on the other hand has its users employ the [[command line]] and [[Python|What is Python?]] to simply <a href="http://programminghistorian.org/lessons/intro-to-the-zotero-api#installing-libzotero"> install Zotero on the computer</a>, no link-clicking or webpage-visiting required. . .. ... Now, while I've got your attention and you're busy downloading, I should let you know that you'll also need a plain text editor if you don't already have one (which you most certainly should by this point). I use <a href="https://www.sublimetext.com/3"> Sublime Text 3</a> on my Windows computer, but you could always try <a href="http://www.vim.org/"> Vim</a> or whatever. . .. ... But [[when and why would I use Zotero|When and why would you use Zotero]]? [[How do I learn to use Zotero]]?

How do I learn to use Zotero? Well, if you're just looking to use the program simply and "manually," no worries. Go check out the <a href="https://www.zotero.org/support/quick_start_guide"> program's quick start guide</a> and just play around (assuming you've [[downloaded it|How do I download Zotero]] already). Let me know once you've finished. . .. ... .... ..... ...... ....... ...... ..... .... ... .. . How'd it go? Enjoy yourself? Can this help you in your work and in organizing your sources, citations, libraries, etc.? Do you want to try something even cooler (and much more complex) with it? [[It went well. I did enjoy myself, thanks for asking. Zotero is my new library organization tool, thanks for showing me! Yes, bring on the next step|It went well. I did enjoy myself, thanks for asking. Zotero is my new library organization tool, thanks for showing me! Yes, bring on the next ste]]! [[It went horribly. It was the opposite of fun. It's not very useful. Hell no, I don't want to keep going]]!

There are so many more ways to organize, sort, store, and archive your data, information, work, and bibliographic information that are beyond this twine and frankly beyond me. There is a lot to be said for those who have mastered the [[command line]] and its abilities (some which are touched upon by the Programming Historian's tutorials to which we have linked) as well as number of other tools out there. I have glommed onto Pandoc & Zotero because I understand them, they fit my work needs, and I feel like they would be helpful for an historian just starting out with digital tools, like you. [[Go back|bibliographic organization with Brad]].

As for tutorials, you may want to check out the Programming Historian’s tutorials on <a href="http://programminghistorian.org/lessons/getting-started-with-markdown"> Markdown</a> & <a href="http://programminghistorian.org/lessons/sustainable-authorship-in-plain-text-using-pandoc-and-markdown"> Sustainable Authorship</a> which have been enormous influences on the writing of this section of the Twine. But if you're looking to test out your skills and need a structured exercise, Professor Shawn Graham has published a <a href="http://workbook.craftingdigitalhistory.ca/Open-Access-Research/Exercises/#exercise-1-learning-markdown-syntax-with-dillingerio-or-proseio"> simple tutorial</a> that has you get down and Markdown. [[Back to basics|Basics for Digital Historians]]. Move on the [[Projects for Digital Historians section|Projects for Digital Historians]] to continue learning how you can [[future-proof|Future-proofing, Sustainable Authorship, and Plain Text]] your work and [[convert your work written in Markdown into other file formats|Pandoc]].

Apart from the useful feature of conversion, Pandoc plays a vital role in our efforts to [[future-proof, control, and disseminate|Future-proofing, Sustainable Authorship, and Plain Text]] our work. Assuming you are on board with this, you should be thinking of Pandoc as your Markdown to publishable file format translator. That is to say that with Pandoc you can render your .md files into a litany of other beautiful formats. Need to share a your Resume.md with a silly employer who only accepts .docx files? Perfect, just convert it. Need to share your syllabus with students in a .pdf? Perfect, just convert it. As well, with Pandoc as your digital typesetting tool, you can add figures, a bibliography, formatting, and easily change citation styles, all using the plain text files you've learnt to use back in [[basics|Basics for Digital Historians]]. Yeah, but [[I don't write in Markdown and I'm not here for sustainable authorship]]. Great! [[How do I download Pandoc]]? Actually, I've already downloaded it. [[How do I learn to use Pandoc]]? [[Go back|Projects for Digital Historians]].

You will need to go through the slightly convoluted Programming Historian <a href="http://programminghistorian.org/lessons/sustainable-authorship-in-plain-text-using-pandoc-and-markdown"> tutorial</a> on [[Markdown]], [[Sustainable Authorship|Future-proofing, Sustainable Authorship, and Plain Text]], and Pandoc. You went through it? Perfect! There ya go, you're a master. You can go now! Be Free! Nothing to see here. What are you doing... Why are you scrolling down here? Please don't look over here! . .. ... But wait! Don't click on that tutorial!!! I didn't tell you before, but I've got a [[secret]]... What? You don't care about my [[secret]]? You've already learnt most of Pandoc with that tutorial? You're finished with me and want to get out of here? Ok, well, I guess you can [[head back|Projects for Digital Historians]] then.

That <a href="http://programminghistorian.org/lessons/sustainable-authorship-in-plain-text-using-pandoc-and-markdown"> tutorial</a> you may or may not have clicked on? It has been a huge inspiration for most of my contributions to this Twine. In fact, as is most of this Twine, my contributions are a little deriative of the Programming Historian. It's unfortunate to say, but I felt you should know. However, I did it to save you some trouble. It isn't always easy going through that <a href="http://programminghistorian.org/lessons/sustainable-authorship-in-plain-text-using-pandoc-and-markdown"> particular tutorial</a> because it just has so much to say. So instead of going through it, beginning-to-end, I've collected the pertinent sections. This way you need not scroll through, searching for the <a href="http://programminghistorian.org/lessons/sustainable-authorship-in-plain-text-using-pandoc-and-markdown#markdown-basics"> Markdown basics when it comes to Pandoc</a>. Or for pointers on how to [[command line]] <a href="http://programminghistorian.org/lessons/sustainable-authorship-in-plain-text-using-pandoc-and-markdown#getting-in-touch-with-your-inner-terminal"> when it comes down to using Pandoc</a>. Or for <a href="http://programminghistorian.org/lessons/sustainable-authorship-in-plain-text-using-pandoc-and-markdown#using-pandoc-to-convert-markdown-to-an-ms-word-document"> instructions on how to Pandoc in general</a>. Or for how you can use Pandoc in combination with your <a href="http://programminghistorian.org/lessons/sustainable-authorship-in-plain-text-using-pandoc-and-markdown#working-with-bibliographies"> bibliographies</a> and [[Zotero]]. [[Go back|Projects for Digital Historians]].

What is the command line? The command line is basically a way to interact with a computer and its programs, allowing you to issue commands to achieve certain functions. It tends to allow for more control than a Graphical User Interface (GUI), which are the more commonly used programs through icons and visual indicators, like the icons on your desktop. There are certain programs that are designed only to be used through the command line, which means knowing how to move around it will open up more options in terms of digital tools for your research. Through the command line one of the things you can do is issue individual commands. Some of the simpler, everyday commands, for Windows, that you can use include: ls: a list of all the files in a directory cd: to move back one directory cd..: to move back to the beginning of the directory cd folder-name: to move to the specific folder requested mkdir: I learned about these commands through tutorials such as this <a href ="http://praxis.scholarslab.org/scratchpad/bash/"> one </a> from the Praxis Program, by Scholars Lab. I recommend checking it out, as it provides a more detailed explanation of these commands. [[go back to basic skills|Basics for Digital Historians]] [[When should I use the command line?]] [[How do I get the command line on my computer?]] [[Why should I use the command line?]] [[I get command line now, skip ahead|Keep Going command line]]

When Should I use the Command Line? Depending on who you talk to, the answer may be that you should ALWAYS use the command line. Theoretically, it is possible to use the command line for virtually all of your computer needs. For a really interesting read about the benefits of using the command line for everything, check out <a href ="http://stephenramsay.us/2011/04/09/life-on-the-command-line/"> this article by Stephen Ramsey </a>. I’m guessing though, for the majority of people reading this, that you will not be using the command line for everything… and that’s OK. On my part, I tend to use the command line only for programs that need to operate on the command line, such as programs like [[Wget]], as well as any program that needs [[python|What is Python?]]. [[What is the command line?]] [[How do I get the command line on my computer?]] [[When should I use the command line?]] [[go back to basic skills|Basics for Digital Historians]] [[I get command line now, skip ahead|Keep Going command line]]

Why Should I use the Command Line? Command line is incredibly useful when working on digital history projects. Not only do many programs require you to use command line, but even the ones that have a GUI (Graphical User Interface) option are able to be better controlled and customized through the command line. In terms of the command line and its customization benefits, something that is really useful with the command lines is the ability to use and write scripts. Scripts are relatively long pieces of command that allow you to tell the computer to do something exactly as you want it down. For example, if you were looking to [[grab a lot of material|data mining]], you could write a script that allowed you to say what specifically you wanted, how fast you wanted it to download, and what to do if it ran into a problem. This allows you to control what you will specifically get. You would write the commands into a [[text editor|Oh no! What do I do? How do I save future-me from certain destruction?]] and then open the text file in the command line to run it through a program like [[python|What is Python?]]. This whole script writing thing might sound scary. To be honest, writing a script from scratch does take a lot of time and skill (I definitely couldn’t do it). What makes this useful though is that often you can find a script that someone else has written for a similar project and tweak it to fit your needs. The scripts that often pop up in Programming Historian tutorials, for example. A really good example of the steps of going through writing a script can be found in <a href ="http://programminghistorian.org/lessons/data-mining-the-internet-archive"> here </a> [[When should I use the command line?]] [[What is the command line?]] [[How do I get the command line on my computer?]] [[I get command line now, skip ahead|Keep Going command line]] [[go back to basic skills|Basics for Digital Historians]]

How Do I get the Command Line on my Computer? So the good news is that you already have this tool on your computer, the (perhaps) bad news is that there are a couple of different options that you will have to choose from. The most basic option is the Command Prompt. You can find this program by looking through the Programs option through the Start icon. The Command Prompt is fairly straightforward and will allow you to run the commands mentioned earlier as well as programs, like [[python|What is Python?]], and scripts. Another option that you will see on your Windows machine is something called Powershell. Powershell is similar to the Command Prompt for the most part, but differs in a couple key ways. The biggest difference is that Powershell is trying to run almost like a Unix machine, but is still running on a DOS machine, which can cause problems. This can also be found through Programs. In personal experience, Powershell makes following tutorials, whether designed for Apple products or Windows, even more of a headache, and I found that it is better to just use the Command Prompt to minimize issues. If you are tired of running into tutorials that you can’t follow along because they assume a Mac user, and you have a PC, you can also consider downloading a program called Git Bash. Git Bash is a kind of downloadable command line and will, in theory, allow you to follow Apple-focused tutorials because it follows Unix, rather than Windows’ system of DOS. If you want to go through a tutorial of downloading and using Git Bash, check one out <a href ="http://programminghistorian.org/lessons/intro-to-bash"> here </a>. A note about Git Bash: Although I really like the idea of it, after downloading it on my machine, I have constantly been having problems getting it to work properly. It may be easiest, for the time being, to try to get away with using Command Prompt until you become more comfortable with the process of command lines. [[When should I use the command line?]] [[What is the command line?]] [[Why should I use the command line?]] [[go back to basic skills|Basics for Digital Historians]] [[I get command line now, skip ahead|Keep Going command line]]

So now you have a beginner’s knowledge of the command line, both in terms of uses, commands and the options for Windows users. Hopefully this introduction has been helpful. If you still feel a little shaky, don’t worry… you will be able to practice and learn more while you go through and learn to use different tools. If you want to check out other basic skills, including the command line program python, [[click here|Basics for Digital Historians]]. If you want to apply your new knowledge of the command line, check out our [[Projects for Digital Historians]].

This tutorial is going to explain how WGET is useful for beginner digital historians, or anyone looking to download large quantities of data quickly. What is Wget? - WGET is a program that allows you to download a mirror/exact copy of a website to your computer quickly. There is no need for downloading individual files or webpages as the programme does it for you. [[Why is Wget useful for historians?]] [[How do I download it and use it?]] [[What can I do once I have the data on my computer?]] Go back to the [[Start]] Go bacl tp [[Projects for Digital Historians]]

What is Data Mining? Data mining is the process of extracting information from a digital source, like the internet archive, and transforming it into an understandable structure so that you can use it in the future. [[When should I use data mining?]] [[How do I set up data mining with the Internet archive?]] [[Why should I use data mining?]] [[go back to projects|Projects for Digital Historians]] [[I'm good with data mining, skip ahead|Keep Going data mining]]

When should I use Data Mining? You should use data mining if you have all the necessary tools at your disposal to do so and there is data online that you want to access, both in terms of data and metadata, for your research. In the example used by the <a href ="http://programminghistorian.org/lessons/data-mining-the-internet-archive"> Programming Historian </a>, we have a python module in order to data mine the internet archive, as well as the script you need to grab the data. If you do not have at least the basic script that you can tweak as needed, and if you do not have a python module already, it would be extremely difficult to go about data mining with the information we have in this discussion. It could be that there are other tutorials out there that could help, or perhaps another tool would be better. [[Why should I use data mining?]] [[What is data mining?]] [[How do I set up data mining with the Internet archive?]] [[go back to projects|Projects for Digital Historians]] [[I'm good with data mining, skip ahead|Keep Going data mining]]

Why should I use data mining? It can be useful to have a script grab a large amount of data for you, rather than doing so by hand. In the <a href ="http://programminghistorian.org/lessons/data-mining-the-internet-archive">Programming Historian tutorial </a>, for example, we end up grabbing thousands of letters from the Internet Archive database. Without tools like this, it would take a lot more time to develop this library of data by hand. A downside of tools like this is the amount of knowledge needed at times to apply it. In this case we have a handy script already set up for us, but the act of writing this script from scratch would be beyond many. The good thing about the script provided is that we can take it and tweak some of the commands to alter what data we want to collect and from which collection. We can use the framework set up for our own specific research needs. Check out our discussion on the [[command line]] to get some more info on scripts. Besides the possible difficulties in writing the needed scripts, there are also some interesting research implications for tools like data mining. If you are interested in reading about these implications, [[click here|Long Answer Data Mining]]. [[How do I set up data mining with the Internet archive?]] [[What is data mining?]] [[When should I use data mining?]] [[go back to projects|Projects for Digital Historians]] [[I'm good with data mining, skip ahead|Keep Going data mining]]

How do I set up Data mining with the internet archive? To data mine the Internet Archive, you will need to use the command line. If you don’t know how to use the command line, we have an intro to that, which can be found [[here|command line]]. Once you are in the command line, you will need to use a program called python to run a specific module designed for working with the internet archive. If you don’t have python, or don’t know how to use it, check out our intro to python [[here|What is Python?]]. Once you are in the python interpreter, you will need to install the module pip and then use that module to install the internet archive and pymarc modules. You will then build a script, also discussed in our command line discussion, to take the data you want from the internet archive. A detailed step by step guide to this process can be found in <a href ="http://programminghistorian.org/lessons/data-mining-the-internet-archive"> the Programming Historian tutorial </a>. Bonus Tip: If you check out the Programming Historian tutorial, note that it says to type the commands starting with “sudo”. “sudo” is a Mac specific command, Windows users need to remove that part of the command for it to work. [[What is data mining?]] [[When should I use data mining?]] [[Why should I use data mining?]] [[Go back to Projects|Projects for Digital Historians]] [[I'm good with data mining, skip ahead|Keep Going data mining]]

You now hopefully understand the process of data mining through sites such as the internet archives a little bit better. Tools like this can be a little trickier to adapt in applying to specific research interests, but hopefully this introduction will give you the confidence to test it out. Feel free to try out other methods of [[collecting data with Sara]]’s discussion, or move ahead to see how to tidy up your data so you can better understand it. Don’t forget to check out our [[basic skills section|Basics for Digital Historians]], with tips for working on the command line, scripts, and python. If none of this is what you are looking for, maybe check out Brad’s discussion on [[bibliographic organization|bibliographic organization with Brad]].

When should I use AntConc? You should use AntConc if you are interested in finding recurring phrases, grammatical patterns. You can also use it to conduct keyword-in-context searches, which allows you to see patterns around a particular word. You can also use AntConc to compare different sets of data with each other. AntConc is useful with all of this, without the need to deal with python and building specific scripts. If any of this applies to your research, AntConc is a good tool to use. [[What is AntConc?]] [[How do I set up AntConc?]] [[Why should I use AntConc?]] [[Go back to Projects|Projects for Digital Historians]] [[I'm good with AntConc, skip ahead|Keep going AntConc]]

Why Should I use AntConc? Short Answer: AntConc is a useful program to examine a large amount of textual data, with the added benefit of being fairly straightforward and easy to use. Something to keep in mind with any analysis like this is that you need to know what you are trying to answer in your analysis; this will shape the way that you use this tool. Long Answer: As you can probably tell, the idea that these tools are not existing in an objective vacuum is a recurring issue that continues to exist with AntConc. If you would like to check out a discussion of the implications of AntConc, check it out [[here|long answer AntConc]]. If not, feel free to keep going. [[What is AntConc?]] [[How do I set up AntConc?]] [[When would it be a good time to use AntConc?]] [[Go back to Projects|Projects for Digital Historians]] [[I'm good with AntConc, skip ahead|Keep going AntConc]]

How do I set up AntConc? It is very easy to set up AntConc. The first thing you need to do is download the program, which can be found at <a href ="http://www.laurenceanthony.net/software/antconc/"> this link </a>. Download the Windows version. Once you have it downloaded, import your data. Make sure that your data is in .txt format, as AntConc won’t be able to read it otherwise. A Note about the tabs: You may be looking at the tabs in AntConc and begin wondering about what they do, so I’ve added a little definition for each tab. Concordance: shows the keyword-in-context view. Concordance plot: visualisation of concordance. File view: shows you the whole file so you can see a larger context for your results. Clusters: words that tend to appear together. Collocates: words that are statistically likely to show up together. Word list: all the words in your data. Keyword list: Comparison between two sets of data. [[What is AntConc?]] [[Why should I use AntConc?]] [[When would it be a good time to use AntConc?]] [[Go back to Projects|Projects for Digital Historians]] [[I'm good with AntConc, skip ahead|Keep going AntConc]]

You now have an understanding of how to use AntConc to better understand your textual data. You can now go back and learn about [[MALLET|What is MALLET?]], you can go and check out Sara’s discussion on [[collecting data|collecting data with Sara]], or you can go check out Brad’s discussion of [[organizing your sources|bibliographic organization with Brad]]. Also, even though there weren’t a lot of skills that you need for using AntConc, go check out our [[basic skills section|Basics for Digital Historians]] for discussions on python, command line and more.

The Implications of AntConc The thing about AntConc, and all of these digital history tools, is that they are not objective. They do take the human process out of actually looking through your data, but the choices you make and the questions you are looking to answer influences the results you will get. In the case of AntConc, the files that you used, the data that you compare, even the way you organized the files from the start, will all impact the results you get and the questions it causes you to pose. In using AntConc, the question around stopwords also comes up. Stopwords are words such as “the, it, as, to, them, is” etc. They are words that tend to not have any significance in the text and so are frequently removed from automated analysis so their frequency doesn’t get included in the results. In the <a href ="http://programminghistorian.org/lessons/corpus-analysis-with-antconc"> Programming Historian tutorial </a>, the stopwords are left in, however you can remove them as well. The decision to remove stopwords will impact the results you get. When you take these words out you are changing the way that the computer will read your data, and can cause you to miss interesting connections with these words. While many people do remove them, and it is fine to do so, it is important to acknowledge how this will influence your results and to be transparent in their removal. Listing the stopwords removed as part of your methodology is one way to do this. [[What is AntConc?]] [[When would it be a good time to use AntConc?]] [[How do I set up AntConc?]] [[Go back to Projects|Projects for Digital Historians]] [[I'm good with AntConc, skip ahead|Keep going AntConc]]

A Note about Digital History Tools So before you get started working with all of these lovely tools it’s important to ask yourself, is this the best tool for what you want to do? There are a lot of tools available and they will all impact your research in different ways. Obviously, a major factor in what tool you are going to use will depend on what tools you know how to use. Once you learn about a couple different tools, as you will going through these projects, you will hopefully have some different options to choose from. Things to think about in terms of what type of tools to use include: What type of data are you using/want to use? How do you want to represent you data? Is this tool the best option for what you want to do? It’s best to make sure that the tool you use is not just the one you know the best, but that it actually reflects the data that you have/want and the project that you are trying to approach. Take this Twine, for example. Are there other ways that we could have organized and presented the information for this project? Yes. So why did we use Twine? In the end, we felt as though the format of Twine was most suited for the project we wanted to end up with. We wanted this project to reflect an almost “choose your own adventure” way of learning about these tools. We wanted people to be able to go in and only learn the things they wanted, and be able to skip past the things that they didn't want/need. We also want the experience to be relatively straightforward. After struggling through tutorials on our own, we wanted to create an experience that got rid of some of the difficulties we dealt with. Basically, we want people to be able to customize their experience, depending on what they were looking for, and to do so easily. [[Go to Projects|Projects for Digital Historians]]

If you haven’t begun to future-proof your work, that is alright. Pandoc can still be useful for you. While you will need to know a little bit about how to use [[Markdown]], it doesn't have to feature prominently in your work, just the files you end up needing to convert. Of course, going the Markdown route is highly recommended though... And just as the Programming Historian and my colleagues suggest throughout this Twine, there is plenty more that can be done with this and all the other tools than just what we or they have suggested. It is best to tailor your use of Pandoc to your own style and environment. Fine, [[just let me download it already|How do I download Pandoc]]. Jeez, I just want to learn [[how to Pandoc|How do I learn to use Pandoc]] already. I'm tired of Pandoc and I just don't think it will be useful for my work. [[Get me out of here|Projects for Digital Historians]].

The big super secret problem with the Programming Historian The Programming Historian can be kind of neglectful to us Windows-users. Their very <a href="http://programminghistorian.org/lessons/intro-to-the-zotero-api"> first Zotero tutorial</a> doesn't work!!!! Often times it can be fairly difficult getting through the tutorials if you have no background or knowledge of the things they walk you through. But I when I first went through this one, it was a whole other ballgame. So fortunately for you, I'm here to save the day with a few magical tips to allow to get through it. Follow along on here as you complete that tutorial. """"""""""""""""""" Alright, so after installing Zotero. Go head off to the [[command line]] and get into [[Python|What is Python?]]. [[Using scripting|Why should I use the command line?]], I copy/pasted "`#make the libZotero library available from libZotero import zotero`" into a .py text file for python to go get, using the following command: "`python -i zotero.py`" From there you want to enter "`python -m pip install win_unicode_console`" into the command line. This is a magical code that will let your Windows computer get the job done. More precisely, it will allow your Windows-tainted command line to understand the specific programming language/dialect being used... After a successful install you will go into your Python interpreter with the simple command "`Python`" Next you will want to import and enable that magical code, first with "`import win_unicode_console`" and then `win_unicode_console.enable()`" So with Unicode helping your machine understand, you can follow the next series of steps. They are a slightly different process from what is outlined in the <a href="http://programminghistorian.org/lessons/intro-to-the-zotero-api"> Programming Historian tutorial</a>. There are a few nuances not mentioned in the tutorial that I can [[show you]].

Great! Since you've got all that in the bag, you can start trying the more sophisticated stuff and really get Zotero working for YOU. The Programming Historian offer a series of tutorials: <a href="http://programminghistorian.org/lessons/intro-to-the-zotero-api"> an introduction</a>, <a href="http://programminghistorian.org/lessons/creating-new-items-in-zotero"> a method</a> for adding new items, and finally a way of <a href="http://programminghistorian.org/lessons/intro-to-the-zotero-api"> counting the frequency of words</a> in your library. This last one is similiar to what Kelly has you doing on her end: counting & analysing data These can be a little time-consuming taken together, but very helpful. But, I suppose I should let you know about the [[big super secret problem with the Programming Historian|The big super secret problem with the Programming Historian]] though, before you get too far into the first tutorial... Having [[trouble with the second tutorial]]? Can't [[complete the third tutorial]]? Also, there's [[more]]... [[Go back|bibliographic organization with Brad]].

JEEZ. [[SCRAM|Zotero]]!

The Implications of Data Mining Besides the issues surrounding the knowledge and confidence needed to use things like scripts, and command line programs like python, there are other implications to tools like data mining. Data mining, like other data collection tools involves issues the external influences surrounding this gathering of information. Even something as simple as a Google search is heavily influenced by the ways in which the search engine interprets your keyword(s) and the results that it offers you based on that interpretation. Similarly in using tools like data mining, as well as other tools like [[SPARQL]] and [[Wget]], you are limited and influenced by the databases that offer the ability to use these tools, as well as the assumptions and interpretations of things like keywords and the organization of data within those databases. Keep in mind that when we use search engines and databases, that they influence our research in interesting ways that can be easy to overlook. [[What is data mining?]] [[When should I use data mining?]] [[How do I set up data mining with the Internet archive?]] [[Go back to Projects|Projects for Digital Historians]] [[I'm good with data mining, skip ahead|Keep Going data mining]]

Who is this tutorial for? This tutorial is to introduce you to SPARQL. SPARQL is the programming language used to navigate RDF (Resource Description Database) databases, which archive an institutions’ collection online. This will give you an overview of SPARQL and a link to the Programming Historian's tutorial where you can download it and try it for yourself. [[Why is it useful?]] [[What do I do now?]] Go back to [[Projects for Digital Historians]] Go back to the [[Start]]

Why is Wget useful for historians? This program could be useful to historians who are looking to download a lot of archival pages or information at once. If you are gathering multiple archives or witness testimonies for a project, or if you are looking to download an entire museum’s collection or files related to specific collections, WGet allows you to simplify this process and save you time. Professionally, this program might be useful for museum officials, archivists, and genealogists. Historians who study the First or Second World War and are looking through large quantities of attestation papers or diary pages may also find this helpful as you can download and group all the papers together easily and at once. Micro-historians who are looking to map and analyse a place over a long period of time may also find this tool useful if they are wishing to download documents, journals, and papers relating to a single place over a span of decades or centuries. Historians interested in social history might find Wget useful if they are looking to download large quantities of diaries, letters, and store records to show relationships between people and places. If you’re looking for data, a great place to begin would be <a href="https://archive.org/details/texts">The Internet Archive</a>, or <a href="http://www.bac-lac.gc.ca/eng/Pages/home.aspx"> Library and Archives Canada</a>. [[How do I download it and use it?]] [[Projects for Digital Historians]] [[Start]]

How do I download it and use it? The Programming Historian’s tutorial is perfect for beginners looking to dive into the world of digital history. This tutorial requires Windows users to use the command line. Not sure what that is? Click here to find out how to find and use the [[command line]]. Helpful tips: It is very important to limit how much information your computer is downloading and how fast that information is being downloaded. By not setting a limit, you could crash the website you’re downloading from! Always remember to include the limit rate into your coding! [[Why is Wget useful for historians?]] [[What can I do once I have the data on my computer?]] Go back to the [[Start]] Go back to [[Projects for Digital Historians]]

What can I do once I have the data on my computer? Once you have downloaded the data on your computer after completing the <a href="http://programminghistorian.org/lessons/automated-downloading-with-wget">Programming Historian tutorial</a> you are ready to learn about [[cleaning and analysing data|cleaning and analysis data with me (Kelly)]] with Kelly Already have the information analysed? Click [[here|bibliographic organization with Brad]] to learn from Brad about how to easily cite your sources! [[Why is Wget useful for historians?]] [[How do I download it and use it?]] Go back to the [[Start]] Go back to [[Projects for Digital Historians]]

SPARQL is a useful language for (art) historians, museum experts, and genealogists who are interested in understanding complex relationships between objects, people, places, and events. It is extremely valuable for people looking to download data on “Linked Open Data” databases such as the British Museum, and the Smithsonian who have uploaded their collections online. For museum experts looking to work with provenance data or understand larger connections between objects and their owners which might be useful for historians or people interested in seeing collections donated by specific people, or even how many paintings the British Museum has that are made with oil. Instead of having to search page by page and through images, SPARQL will download all the relevant information Try SPARQL with <a href="http://programminghistorian.org/lessons/graph-databases-and-SPARQL">The Programming Historian</a>. TipThe data set provided is only for the specific tutorial. If you wish to look at different sets of data you will need to tweak the coding. It is recommended that you have some prior experience in coding or try other tutorials before attempting SPARQL, as even I have not mastered the language and the lesson. [[What do I do now?]] [[Projects for Digital Historians]] [[Start]]

What do I do now? Now that you have completed the Programming Historian tutorial, you can play with your data with [[Kelly's tutorials|cleaning and analysis data with me (Kelly)]]. You can also use <a href="http://palladio.designhumanities.org/">Palladio</a>, a network visualisation programme to map your data. This could be interesting to see how objects have travelled or how many pieces of art a museum collects from across the world, or where a collector compiled their collection. However, Palladio can only work with a few hundred sets of data at once, and so it is best to use this mapping service for smaller amounts of information or in order to understand a sample set of data. While there is not a tutorial in this twine about Palladio, The Programming Historian offers one <A href="http://programminghistorian.org/lessons/creating-network-diagrams-from-historical-sources">here</a>. [[Start]] [[Projects for Digital Historians]] [[collecting data with Sara]] [[bibliographic organization with Brad]]

First off, you'll need to know how to use the [[command line]] for this to work. And [[Python|What is Python?]]. And a <a href="https://www.sublimetext.com/3"> plain text editor</a>. And [[Zotero|How do I download Zotero]]... Got it? You should, I've only been asking you about it, like, forever... [[Yes, I've got it. Let's go|How do I learn to use Zotero2]]. [[No, I haven't bothered doing it all|JEEZ]]...

I'm sorry you feel that way. But you are your own master, I won't hold it against you. In fact, I'm curious to know why it didn't go well and/or why you prefer using other methods of organizing your bibliographies. Send me an email about it at [email protected] or tweet me @BradWieb3. Take me back to the [[Start]]. Take me back to [[Projects for Digital Historians]].

No you're not. Digital History is never done. There is always room for more learning and more mastery. [[Start]]

Zotero can also be used <a href="http://programminghistorian.org/lessons/sustainable-authorship-in-plain-text-using-pandoc-and-markdown#working-with-bibliographies"> in combination</a> with [[Pandoc]] when you want to bring your bibliography into your plain text [[Markdown]] documents. You'll be learning about APIs (Application Programming Interface) in those tutorials. My suggestion? Take a look at Professor Shawn Graham's <a href="http://workbook.craftingdigitalhistory.ca/module-2/Exercises/#exercise-3-apis"> workbook</a> which will help with learning about APIs. [[Bring me back to those tutorials, I wasn't finished|How do I learn to use Zotero2]]! [[Go back|bibliographic organization with Brad]].

So following all of that you will need to input the following commands on the command line in the Python interpreter (this must be done right after you finished that Unicode process we just did because it would have to be re-added if you close the Command Line). Bearing in mind that capitalization matters and you should only be getting ">>>" after every command you put in unless otherwise stated, you will type the following: "`from libZotero import zotero`" "*`zlib=zotero.Library('group','155975','<null>','9GLmvmZ1K1qGAz9QWcdlyf6L')`" "`items = zlib.fetchItemsTop({'limit': 5, 'content': 'json,bib,coins'})`" * At this point the tutorial says you should have an output, but you should NOT. `for item in items:` * At this point the tutorial says to include the rest of the command, but it should be included after! "` print 'Item Type: %s | Key: %s | Title: %s' % (item.itemType,item.itemKey, item.title)`" * you need to include a series of 4 spaces before this "`print`" command. That is why I've included that space. A series metadata should appear. "`for item in items:`" * At this point the Tutorial says to include the rest of the command, but it should be included after! "` print item.bibContent`" * Again, you need to include a series of 4 spaces before this `print` command. A series of metadata should appear. #Success [[Go back|How do I learn to use Zotero2]].

When I went through this one, I had some trouble too. After connecting to Zotero's sample library I played around with the CREATE and NAME commands found in the tutorial. From there it gotThis part got a little tricky. I believe all the "print" commands require an indent (i.e. four spaces before actually writing up the command). After inputing those commands you will often get "..." which simply requires that you just press Enter and the "..." should revert back to the Python Interpreter prompt. Hold on though... After I finished doing the tutorial and supposedly adding all my new entries and tags, it appeared that I had been successful. However, I did not receive the desired output, nor did I find my newly created item in my Zotero library through the desktop program... Send me an email at [email protected] if you manage to get it working. **AN IMPORTANT NOTE**: All of this can be made easier if you write a script/sequence (in a .txt file) with all the commands in the correct order and then send Python to go and read and implement it. It is best to try out a tutorial one command at a time to start off, but multiple and regular usage of [[Python|What is Python?]] makes using the [[script|Why should I use the command line?]] method more ideal. [[Go back|How do I learn to use Zotero2]].

I've got to be completely honest with you. I've never gone through this one... Can't help you, `#sorrynotsorry`. [[Go back|How do I learn to use Zotero2]].