A component is a kind of Joomla! extension. Components are the main functional units of Joomla!; they can be seen as mini-applications. An easy analogy would be that Joomla! is the operating system and the components are desktop applications. They are usually displayed in the center of the main content area of a template (depending on the template).
Most components have two main parts: an administrator part and a site part. The site part is what is used to render pages when being called during normal site operation. The administrator part provides an interface to configure and manage different aspects of the component and is accessible through the Joomla! administrator application.
Joomla! comes with a number of core components, like the content management system, contact forms and Web Links.
There are many articles, tutorials, references and FAQs which focus on component development. If this is your first time developing a component for Joomla, you should start with the Absolute Basics of How a Component Functions. If needed, you can visualise the control flow of a component with these diagrams.
Once you have read the tutorial and/or tried the example component, you can focus more on the specifics of your component with additional articles. These are listed on the Component Development Portal or any of the supporting Portals for Development(Plugins, Modules or Templates).