149 lines
6.7 KiB
Markdown
149 lines
6.7 KiB
Markdown
# Fifth Assignment: IMDb Scraper 🎬
|
|
|
|
## Introduction
|
|
|
|
This project delves into the practical application of HTML parsing techniques using Java and the JSoup library. The objective is to extract and manipulate data from a structured HTML document containing information about various movies. Key tasks include parsing the document, creating Java objects to represent the data, sorting the information based on different criteria, and presenting the results effectively.
|
|
|
|
## Why Learn Web Scraping? 📊
|
|
|
|
Mastering web scraping empowers individuals and businesses to glean valuable insights, automate tasks, and make informed decisions based on real-time data.
|
|
|
|
- In e-commerce, web scraping aids in monitoring competitors' prices, tracking product availability, and gathering customer reviews. In finance, it facilitates trend analysis, investment decisions, and risk assessment.
|
|
|
|
- Research, academia, journalism, and media professionals leverage web scraping for data collection, trend tracking, sentiment analysis, and investigative reporting.
|
|
|
|
By mastering web scraping, professionals can extract data efficiently, outperform competitors, and drive data-informed success in diverse industries.
|
|
|
|
## Prerequisites ✅
|
|
|
|
Before diving into this project, make sure you have the following tools and knowledge:
|
|
|
|
### Tools Needed:
|
|
- **Java 17 or higher** : Download the latest version from [Oracle's Java website](https://www.oracle.com/java/technologies/downloads/).
|
|
- **Git** : Install Git, a powerful version control system, from [Git's official website](https://git-scm.com/downloads).
|
|
- **Maven** : Utilize Maven as the build tool. You can find installation instructions [here](https://maven.apache.org/install.html).
|
|
|
|
### Knowledge Required:
|
|
In addition to the tools mentioned above, familiarity with the following topics is crucial for successful project completion:
|
|
- **Jsoup** : Understanding how to access, parse, and manipulate HTML files using Jsoup.
|
|
- **HTML** : Knowledge of various HTML tags, their purposes, and how to identify and manipulate them using Jsoup.
|
|
- **CSS** : Understanding how to style HTML elements, work with CSS classes, and utilize selectors efficiently.
|
|
|
|
By having a solid grasp of these tools and concepts, you'll be well-equipped to excel in this project. 🚀
|
|
|
|
## Objectives 🎯
|
|
|
|
The primary objectives of this assignment are:
|
|
|
|
- Parsing an HTML file with movie data.
|
|
- Extracting relevant information such as title, rating, and release year.
|
|
- Implementing Java classes to represent movies.
|
|
- Utilizing JSoup for HTML parsing.
|
|
- Applying sorting algorithms for organizing data by title, release year, and rating.
|
|
- Displaying the sorted movie information effectively.
|
|
- Passing the provided **JUnit tests** to verify the correctness of your code.
|
|
|
|
## Tasks 📝
|
|
|
|
1. **Set Up the Project:**
|
|
|
|
- Start by cloning the repository to your local machine.
|
|
- Open the project in an IDE such as IntelliJ.
|
|
|
|
2. **Integrate JSoup Dependency:**
|
|
|
|
- Include the JSoup dependency in your `pom.xml` file.
|
|
|
|
```xml
|
|
<dependency>
|
|
<groupId>org.jsoup</groupId>
|
|
<artifactId>jsoup</artifactId>
|
|
<version>1.14.3</version>
|
|
</dependency>
|
|
```
|
|
|
|
3. **Complete the Movie Class:**
|
|
|
|
- Complete the `Movie` class in Java to encapsulate movie-specific data.
|
|
|
|
4. **HTML Parsing using JSoup:**
|
|
|
|
Utilize JSoup to parse the HTML file located in the resources folder at this path: `src/main/resources/Movies.html` and extract relevant information such as:
|
|
|
|
- `<h3 class="movie-title">` = for movie titles
|
|
- `<span class="movie-rating">` = for movie ratings
|
|
- `<span class="movie-year">` = for movie release years
|
|
|
|
**Note: The elements are nested within other tags and div closures. Look out for unneeded text like '/10' inside elements.**
|
|
|
|
Here is a general structure of the HTML you will be working with:
|
|
|
|
```html
|
|
<div class="col-md-4 movie">
|
|
<h3 class="movie-title">The Godfather</h3>
|
|
<div class="movie-info">
|
|
<strong>Rating:</strong> <span class="movie-rating">9.2/10</span><br>
|
|
<strong>Year:</strong> <span class="movie-year">1972</span><br>
|
|
</div>
|
|
</div>
|
|
```
|
|
|
|
5. **Implement Sorting Functionality:**
|
|
|
|
- Develop sorting algorithms for various criteria like title, year, and rating.
|
|
|
|
6. **Run JUnit Tests:**
|
|
|
|
- We have provided tests using JUnit 5 (`ParserTest.java`). Run them via your IDE or by executing the following Maven command in your terminal to verify your logic:
|
|
```bash
|
|
mvn test
|
|
```
|
|
|
|
7. **Visualize Results:**
|
|
|
|
- Clearly present the sorted movie data using the `Main` class you implement.
|
|
|
|
## Notes 📝
|
|
|
|
Here are some important points to keep in mind:
|
|
|
|
- There is a `Help` folder located at the root directory of the project which contains the sorted movies each by different priorities. It is there just so you can see what your output should look like in the end.
|
|
- The unit tests are provided to assist you in understanding the project requirements. Your final grade is not solely dependent on their results; they are meant to aid your learning process. Remember to enable GitHub Actions for the test workflow to run on GitHub.
|
|
- Feel free to leverage ChatGPT for learning web scraping and resolving any challenges that you may not find solutions to on the internet. Utilize its capabilities to enhance your understanding and overcome obstacles effectively.
|
|
|
|
## Evaluation 🧐
|
|
|
|
Your success will be judged based on:
|
|
|
|
- Functionality of the code in parsing HTML, creating Java objects, and sorting data.
|
|
- Readability and maintainability of the code.
|
|
- Efficiency of sorting algorithms.
|
|
- Clarity of the output format.
|
|
- Successful execution of the provided JUnit tests.
|
|
|
|
## Submission 📁
|
|
|
|
To share your work:
|
|
|
|
1. Add your mentor as a contributor to the project.
|
|
2. Create a `develop` branch for implementing features.
|
|
3. Use Git for regular code commits.
|
|
4. Push your code to the remote repository.
|
|
5. Submit a pull request to merge the `develop` branch with `main`.
|
|
|
|
If you have any further questions or need clarification, do not hesitate to reach out to your mentor. Good luck with your project! 🌟
|
|
|
|
## TA Tutorial Videos 🎥
|
|
|
|
To help you succeed in this assignment, our TA team has recorded dedicated tutorial videos covering the essential topics. Please watch these before starting. You can access the folder containing the tutorial videos here:
|
|
|
|
- [TA Tutorial Videos - Movies Folder](https://drive.meshcomp.ir/d/3e149a8347a841f6bb81/)
|
|
|
|
## Additional Resources 📚
|
|
|
|
- [Jsoup - Official website](https://jsoup.org/cookbook/)
|
|
- [Jsoup - Javapoint](https://www.javatpoint.com/jsoup-tutorial)
|
|
- [Jsoup - Baeldung](https://www.baeldung.com/java-with-jsoup)
|
|
- [HTML - W3](https://www.w3schools.com/html/)
|
|
- [CSS - W3](https://www.w3schools.com/css/default.asp)
|